Topic: Continuously updating a page using Ajax.

I'm absolutely stumped on this one... I've got a page that needs to update dynamically as a web scraping script gets information.
How can I adapt what I have so that I get a successful Ajax call? When I debug my app I can see that I'm sending a get request to http://localhost:3000/noko/nokosearch?search_string=foo+bar . This is exactly what I want. 

Here's the script. I threw it in a controller.

class NokoController < ApplicationController

def nokosearch
    j = ActiveSupport::JSON
    keywords = params[:search_string]
    #initialize @product_links as empty array
    @product_links = Array.new

    # initialize hash of arrays
    @products = Hash.new {|h,k| h[k]=[]}

    # make keywords string friendly to use in search url
    keywords = keywords.split(' ').join('+')

    # Initialize an array that will be filled with links to individual products that nokogiri finds for all the products on a page
    noko_links = Array.new

    # URL construction
    base_url = "http://www.amazon.com/s/url=search-alias%3Daps"
    keyword_url = base_url + "&field-keywords=" + keywords

    # Opening up a page from the URL with a list of products. Selecting all the links for the products and removing empties.
    page = Nokogiri::HTML(open(keyword_url))
    @product_links = page.css("a").select{|link| link['class'] == "title"}
    @product_links.compact! # get rid of nils

    # Using the Fletcher module for each link to product pages to get simple information returned as strings (names, descriptions, image urls etc)
    @product_links.each do |link|
      fletchedProduct = Fletcher.fetch link['href']

      @products["name"] << fletchedProduct.name
      @products["description"] << fletchedProduct.description
      if (fletchedProduct.image != nil)
        @products["image"] << fletchedProduct.image[:src]
      else
        @products["image"] << ""
      end
      @products["price"] << fletchedProduct.price

    end
    @jsonproducts = j.encode(@products)
    render :json => @jsonproducts

  end

end

The script takes a really long time to execute before it returns anything. I want to "render" every time at the end of the

@product_links.first do |link|

loop, but I don't know how.

I think I have problems all over the place! This is what I have in the application.js for the client side:

$('#my-search').live('change', function () {
  alert('this is an alert');
    $.ajax({
    // type: "GET",
    url: 'noko/nokosearch',
    data: {search_string: document.getElementById('my-search').value}, 
    dataType: 'json',
    complete: function() {
      alert("Ajax Complete!");
    },
    success: function(data){
      alert('you have success');

        $.each(search_results, function(index, search_result) {
          $('#product-list').append('<li>'+search_result.name+'</li>');
        });
      },
      error: function() {
        alert("Ajax Error !");
      },


  });
});

And I'm not sure if I have the routing right:

get 'noko/nokosearch' => 'noko#nokosearch'

Well, for complete context, all the code is available here on github:

https://github.com/dbwest/infinitestore.git

Help me, please!

Re: Continuously updating a page using Ajax.

It would probably be best to use some sort of worker queue setup to handle this. Using something like http://mperham.github.com/sidekiq/ or https://github.com/defunkt/resque/.

You would need an action on the controller that tells the worker to start processing the search job.  That worker would do the work in the background and drop the results into some sort of datastore as it is doing the work.  Then you would have another action on the controller that you could poll via an AJAX call that would get the latest search results that the worker added to the datastore since the last time the AJAX call was made.