tag:blogger.com,1999:blog-5356161893554749345.post6967623893740600952..comments2023-04-16T07:29:35.987-07:00Comments on FriendFeed Blog: Simple Update Protocol: Fetch updates from feeds fasterBret Taylorhttp://www.blogger.com/profile/04488384301690653632noreply@blogger.comBlogger34125tag:blogger.com,1999:blog-5356161893554749345.post-1917851280504464372009-07-17T04:32:03.400-07:002009-07-17T04:32:03.400-07:00Looks like my last vague idea was not only already...Looks like my last vague idea was not only already developed, but already implemented. Just finished reading a sentence of Louis Gray's post.Mark Esselhttps://www.blogger.com/profile/02148726313106222507noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-74135436377166869152009-07-17T04:12:47.633-07:002009-07-17T04:12:47.633-07:00If I understand thus right we have a network effic...If I understand thus right we have a network efficiency problem and limitations based on data structure. Large providers may have millions or potentially billions of individual nodes that have a status update flag.<br />Current efficient polling is done by comparing their remote time stamp and the node's last update time.<br />Excuse me for the modest description this problem space is new to me (but important to what I'm working on).<br /><br />Is there a viral hub heirarchy? This would be a limited number of remote servers who simply pass along the last update time to all subscribers (and any polling system could also serve last node update time). You would subscribe to a virtual server represented by a dynamic list of your nearest neighbors. A network similar to bittorrent but only concerned with last update time.<br /><br />The storage could get problematic for large databases (even of a simple time) so each serving hub could keep what it needs for internal use and just push all the times to it's short list of clients.<br /><br />I'm trying to think of a decentralized status architecture, would appreciate expert opinionsMark Esselhttps://www.blogger.com/profile/02148726313106222507noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-58188416849514127032008-09-10T05:03:00.000-07:002008-09-10T05:03:00.000-07:00whats wrong with having a simple atom “update” fee...whats wrong with having a simple atom “update” feed listing recently changed feeds? that feed could use RFC5005 and consumers wouldn’t need to worry about polling faster than the SUP feed turns over completelyAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-64377328011336883032008-09-04T13:39:00.000-07:002008-09-04T13:39:00.000-07:00This is cool, but what about the xmpp pubsub? Isn...This is cool, but what about the xmpp pubsub? Isn't that what a lot of people are recommending now? What are your thoughts on this?Chrishttps://www.blogger.com/profile/08539566345676172517noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-31173573004305529042008-09-03T11:52:00.000-07:002008-09-03T11:52:00.000-07:00Here's an extended version of Atomstream.py with X...Here's an extended version of <A HREF="http://pastebin.ca/1192834" REL="nofollow">Atomstream.py with XMPP support</A> (it's buggy and may crash after some time).<BR/><BR/>The script turns ATOM feeds into XHTML-IM messages.<BR/><BR/>Might be fun to add a track capability with Pusbub, though. But I digress.<BR/><BR/>As regards data exchange, XMPP Pubsub looks like an interesting solution, considering for example that contrarily to the SixApart ATOM stream, Friendfeed servers would only receive messages from subscribed XMPP Flikcr accounts, not all the data which would avoid filtering.<BR/><BR/>SUP looks interesting but so 20th century.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-35086760955664967102008-09-03T10:05:00.000-07:002008-09-03T10:05:00.000-07:00But what do you do when you're polling thousands o...But what do you do when you're polling thousands of SUP feeds individually? Obviously you'll need a way to see if any of the feeds in any of the SUPs you are tracking have updated in order to save even more bandwidth. Hmm... maybe we need <I>another</I> protocol ;-)Jesse Hattabaughhttps://www.blogger.com/profile/04251500284884217703noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-91688344255530752482008-09-02T17:55:00.000-07:002008-09-02T17:55:00.000-07:00http://www.aaronsw.com/2002/atomstream/atomstream....http://www.aaronsw.com/2002/atomstream/atomstream.pyAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-51470085932733966552008-08-29T14:38:00.000-07:002008-08-29T14:38:00.000-07:00curl -N -s http://updates.sixapart.com/atom-stream...curl -N -s http://updates.sixapart.com/atom-stream.xml | grep --line-buffered "application/atom+xml" | grep --line-buffered 'rel="self"'Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-38285805784989335822008-08-29T09:07:00.000-07:002008-08-29T09:07:00.000-07:00I'm not sure SUP would help much for a large scale...I'm not sure SUP would help much for a large scale generator of feed-based activity like Netflix without some notion of subscription resulting in SUP feed per feed consumer. The personalized Netflix feeds generate generate about 6 million posts per day (about 2M each of queue adds, shipped DVDs and received DVDs). Given that any given feed consumer is likely only interested in a small fraction of the 8.4M+ subscribers, the signal-to-nose ratio in a general SUP feed would be quite low.Michael Harthttps://www.blogger.com/profile/02337340210503981363noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-24274070109991339662008-08-28T15:34:00.000-07:002008-08-28T15:34:00.000-07:00Mike D, other:Re: multiple databases. First, you c...Mike D, other:<BR/><BR/>Re: multiple databases. First, you could serve multiple SUP feeds corresponding to your multiple databases. Second, if a client like FriendFeed or Gnip is hitting all your feeds, generating a single SUP feed is provably easier on your servers: you could poll your own RSS feeds on the server generating the SUP.<BR/><BR/>re: "In an RSS architecture, a system must generate output for only the users that are actively being polled." SUP is useful when two conditions are satisfied: a lot of your feeds are being hit by a single client, and many of those feeds are not updated during the polling interval. SUP is not designed for the use case you're talking about.<BR/><BR/>Re: "it appears there is no way to actually fetch the contents of an update without knowing the URL that originally led you to a SUP-ID" Yes, that's the point. <BR/><BR/>Re: push solutions. Maintaining and producing/consuming an open connection is harder for both ends.Andrew B.https://www.blogger.com/profile/02903366372812261992noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-46531006911567036382008-08-28T13:00:00.000-07:002008-08-28T13:00:00.000-07:00Atom feeds solved this problem a long time ago, as...Atom feeds solved this problem a long time ago, as evidenced at six apart with http://updates.sixapart.com<BR/><BR/>This solution is neither simple nor does it take into account years of prior art of solving this problem through push.John Adamshttps://www.blogger.com/profile/09456285341916959260noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-5931210755389844222008-08-28T12:33:00.000-07:002008-08-28T12:33:00.000-07:00I like this idea, simple and elegant. I agree that...I like this idea, simple and elegant. I agree that constant polling sucks and is a terrible solution for what services like (friendfeed or Feedheads are doing), however, I struggle to imagine something like this being promptly adopted by most feed producers. <BR/><BR/>Google Reader's shared feed doesn't even make use of IF-Modified-since headers. They're google!!Unknownhttps://www.blogger.com/profile/08391463372142565313noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-76380942278129386982008-08-28T08:56:00.000-07:002008-08-28T08:56:00.000-07:00I'm still having trouble envisioning what a provid...I'm still having trouble envisioning what a provider's implementation of this SUP feed would look like. MikeD makes good points about federated databases making this far more complicated and expensive. Paul Watson rightly points out that this feed would be gigantic. If it were hypothetically to contain 80,000 items, Flickr's would completely turn over every 25 minutes or so.<BR/><BR/>Rather than continuing to complain endlessly, I'd like to hear from FriendFeed themselves how they would go about creating a SUP stream for FriendFeed itself. I want to be convinced that this is a valid easy-way-out of the much longer road of popularizing XMPP.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-19334783405251844122008-08-28T08:13:00.000-07:002008-08-28T08:13:00.000-07:00oh god... this is not a "simple idea that no ...oh god... this is not a "simple idea that no one thought of before". This is weblog.com's changes.xml (courtesy of Dave Winer) all over again.<BR/><BR/>In fact, the inherent scaling problems in such a solution led to the RSS <cloud>-Element which attempted to solve the problem at the source, but through XML-RPC, which is why the cool kids don't support it. <BR/><BR/>(That and that it breaks if you get 1000s of timeouts from unavailable RPC endpoints)<BR/><BR/>Also: what Mike D said in his 2nd post.<BR/><BR/>His first one missed the point. HTTP HEAD is not going to solve this problem.<BR/><BR/>Please take this back to the drawing board. You're right, this problem needs a solution.<BR/><BR/>But the solution will not be simple, because it's a really complex problem.<BR/><BR/>Does FriendFeed even run a ping server?Unknownhttps://www.blogger.com/profile/06368456392029735365noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-61545818006903182142008-08-28T07:35:00.000-07:002008-08-28T07:35:00.000-07:00Is there a document somewhere with the format of t...Is there a document somewhere with the format of the <link> tags and the SUP feed? Examples aren't enough.Finhttps://www.blogger.com/profile/09388247167019618091noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-55264261616700549512008-08-28T06:08:00.000-07:002008-08-28T06:08:00.000-07:00So Flickr would have one SUP feed that lists any r...So Flickr would have one SUP feed that lists any recently updated feeds? That could be 80,000 items, right? Is that going to scale?Paul M. Watsonhttps://www.blogger.com/profile/05397150081995980133noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-63716885879178502012008-08-27T23:59:00.001-07:002008-08-27T23:59:00.001-07:00Guys, what do you think of Atom streams: http://up...Guys, what do you think of Atom streams: http://updates.sixapart.com/ ?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-84836799732941734102008-08-27T23:59:00.000-07:002008-08-27T23:59:00.000-07:00Your first and third benefits incorrectly assume t...Your first and third benefits incorrectly assume that all information resides in a single database, which is not the case for any site large enough to benefit from your proposal. Most large architectures work on the concept of sharding. Groups of systems that handle a smaller subset of the overall user population, without a need to know about each other. Unless I am misunderstanding some vital component, what you are asking providers to do is create a single point of aggregation that must know about all updates within all shards. This in itself creates a far more complex engineering problem than an end-site making 200 requests a second (which remember, in a sharded architecture, scales by adding a few extra queries each to a number of autonomous systems).<BR/><BR/>SUP would also multiply the volume of raw computing work that needs to be done by a data provider. In an RSS architecture, a system must generate output for only the users that are actively being polled. In a "SUP" system, resources must now be spent to generate output for all users, regardless of if anyone cares to come looking for it. Your claim that it reduces load on both ends is false, the only reduction in work is on the consuming end.<BR/><BR/>Your claims of not exposing secret feed URLs is also incorrect, as it appears there is no way to actually fetch the contents of an update without knowing the URL that originally led you to a SUP-ID.<BR/><BR/>I do not dispute that it works over HTTP, or that it is compact.Mike Dhttps://www.blogger.com/profile/14057099816957047303noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-63165735607388565362008-08-27T22:56:00.000-07:002008-08-27T22:56:00.000-07:00I don't see why it would not work, but still, it f...I don't see why it would not work, but still, it feels like another layer of duct tape.<BR/><BR/>Why insist on HTTP? Those 43 services that you pool constantly are the one who have the engineering resources to do this right.<BR/><BR/>Even Livejournal did it better 2 or 3 years ago with the TCP-based ping stream.<BR/><BR/>I understand that you have a problem to solve and you are looking for a quick way out, but really, do it right and move to a stream based solution, TCP, XMPP, other, take your pick.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-62496690310502903362008-08-27T21:29:00.000-07:002008-08-27T21:29:00.000-07:00Very simple and clever. Friendfeed-like!Very simple and clever. Friendfeed-like!Edwin Khodabakchianhttps://www.blogger.com/profile/13744784219447329954noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-86947874176215397832008-08-27T21:06:00.000-07:002008-08-27T21:06:00.000-07:00I like this very much!Did you think of using the e...I like this very much!<BR/><BR/>Did you think of using the existing RSS format for the list of feeds that are updated instead of the new json SUP format?Yaarhttps://www.blogger.com/profile/09280567613091088643noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-24934937720505784902008-08-27T19:27:00.000-07:002008-08-27T19:27:00.000-07:00Probably because spitting a feed of updated items ...Probably because spitting a feed of updated items is a 20 minute job, while implementing XMPP would be a major PITA.Rogerhttps://www.blogger.com/profile/05124842676463037949noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-24236694661515563442008-08-27T18:45:00.000-07:002008-08-27T18:45:00.000-07:00Here's my comment from the Read Write Web article:...Here's my comment from the Read Write Web article:<BR/><BR/>"""<BR/>"On July 21st, 2008, FriendFeed crawled Flickr 2.9 million times to get the latest photos of 45,754 users of which 6,721 of that 45,754 visited Flickr in that 24 hour period, and could have 'potentially' uploaded a photo."<BR/><BR/>Source: http://www.slideshare.net/kellan/beyond-rest (Slide 16)<BR/><BR/>RSS is simply not going to get us there.<BR/>"""<BR/><BR/>Given that this still requires the services y'all are scraping to Do Something, why not become the killer-app for XMPP?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-84945428117492376312008-08-27T18:26:00.000-07:002008-08-27T18:26:00.000-07:00sounds a bit like newnews for nntp did to help new...sounds a bit like newnews for nntp did to help news traverse to the various nntp servers faster. Here's a crazy thought maybe a new http verb "poll" in combination with if-modified-since. if used against a single feed it would limit the transmitted items only to new items. If used against a domain it could return modified pages/feeds for that domain. It could even increase the speed for crawlers for search engines. the accept header could even limit what types of updates are returned via poll.Shawn McCollumhttps://www.blogger.com/profile/05949614261519543610noreply@blogger.comtag:blogger.com,1999:blog-5356161893554749345.post-45146587652853672242008-08-27T18:17:00.000-07:002008-08-27T18:17:00.000-07:00This is one of those simple ideas that one wonders...This is one of those simple ideas that one wonders why no one thought of before. It's a good proposal and a required one in the rapidly growing aggregation/Lifestreaming world. I am sure the proposal will be widely and quickly SUPported.Anonymousnoreply@blogger.com