Copying from NYTimes API JSON to a local JSON file

Processing Coding Questions
1

Hi all,

I really appreciate if you take the time and help me solve this issue as I have spent hours and haven’t made any progress.

What’s the project
I’m creating a data visualization project using New York Times Article API and Processing. The idea is to show all images published in NYTimes with a certain keyword. Then the user hovers over each image and gets to see the information about that specific images such as the year, the page it was published and etc.

What’s the problem
With every request, NYTimes API returns only 10 results (not 10 images but 10 articles). In order to be able to show all images in the sketch I need to have a JSON file with all the images’ URLs.

What I’ve tried
I figured maybe making a master JSON file helps. I can copy the first 10 results, then I’ll go to the second page (through a for loop) of the NYT JSON and copy the next 10 results and so on until there are no more articles.
So far I have looked into these functions: saveJSONObject, saveJSONArray, arrayCopy(); and concat();

Questions

  1. Have you had a similar experience with NYTimes API? How have you solved this problem?
  2. Is it possible to merge JSON files through Processing?
  3. Is there another way to do the same thing?

Here’s the code

String apiKey = "PUTYOURAPIKEYSTRINGHERE";
String url = "https://api.nytimes.com/svc/search/v2/articlesearch.json";
String query = "?query=iran";
int page = 0; // each page shows 10 results
String beginDate = "&begin_date=20080101";
String endDate = "&end_date=20181030";
//String filter = "&fl=multimedia";
//String nyURL = "https://www.nytimes.com/";
//String[] multimedia = new String[900];
//String imageURL;
JSONObject json; //nytimes api json
JSONArray master; // the local json file that will be the master json file
JSONArray test; // another local json file, I want to copy every 10 new search results into and then copy this array to the master array, if I understood it well processing differentiates between arrays and json arrays

void setup() {
  json = loadJSONObject(url+query+"&page="+page+beginDate+endDate+"&api-key="+apiKey); 
  for (int i = 0; i < json.getJSONObject ("response").getJSONArray("docs").size(); i++) { // look at all objects in "docs"
    //json = loadJSONObject(url+query+"&page="+page+beginDate+endDate+"&api-key="+apiKey);    
    JSONArray mm = json.getJSONObject("response").getJSONArray("docs").getJSONObject(i).getJSONArray("multimedia"); // find the "multimedia" array in each object
    if ( json.getJSONObject("response").getJSONArray("docs").getJSONObject(i).getJSONArray("multimedia") != null) { //if multimedia is not null
      saveJSONArray(mm, "data/test.json"); // save multimedia content into test.json
      JSONArray test = loadJSONArray("data/test.json"); // load test.json
      if(page==0){ 
      arrayCopy(test,master); //copy whatever's in test to master
      }
      //println(page);
      page = page + 1; 
      if (page != 0) { // if page is not 0, meaning there are already some stuff written in the master file
        //arrayCopy(test,0,master,69,test.size()); // I wrote 69 beacuse I know page 0 includes 68 indexes
        //master = concat(test, mm); 
        println(master.size());
      }
      //saveJSONObject(json, "data/test.json");
      ////master = loadJSONObject("data/test.json");
      //for (int page = 0; json != null; page += 1) {
      //  master = concat(json,master);
      //  print(master);
      //}
    }
  }
}
1 Like
2

sorry, i try WITHOUT the page thing,
and this master ( array of all multimedia links ) looks already good,
made by manual concat…

yes you can show all pictures with it,
but i doubt that this is what you really need,
as you loose the reference to the articles
and you just flatten out the structure

String url = "https://api.nytimes.com/svc/search/v2/articlesearch.json";
String query = "?query=iran";
int page = 0; // each page shows 10 results
String beginDate = "&begin_date=20080101";
String endDate = "&end_date=20181030";
String apiKey = "PUTYOURAPIKEYSTRINGHERE";

JSONObject json; //nytimes api json
JSONArray master; // the local json file that will be the master json file
String masterf = "data/master.json";

String nyURL = "https://www.nytimes.com/";
String imageURL;
PImage img;

boolean dprint = true;  //  if (dprint) println(json);
String test1 = "data/json.json";
String test2 = "data/mm.json";

void get_nytimes_json() {
  json = loadJSONObject(url+query+"&page="+page+beginDate+endDate+"&api-key="+apiKey);
  if (dprint) println("json save to file "+test1);  
  if (dprint) saveJSONObject(json, test1 );
  for (int i = 0; i < json.getJSONObject ("response").getJSONArray("docs").size(); i++) { // look at all objects 
    JSONArray mm = json.getJSONObject("response").getJSONArray("docs").getJSONObject(i).getJSONArray("multimedia"); // find the “multimedia” array in each object
    if ( mm != null) { //if multimedia is not null
      if (dprint) println("mm save to file "+test2);  
      if (dprint) saveJSONArray(mm, test2);                       // you will later only see the last one !!
      for ( int k=0; k<mm.size(); k++) {
        JSONObject json_mm_rec =  mm.getJSONObject(k);
        if (dprint) println("json_mm_rec "+k+" "+json_mm_rec);          
        master.append(json_mm_rec);
      }
    }
  }
  saveJSONArray(master, masterf);
}

void setup() {
  size(500, 500);
  master = new JSONArray();   // init
  get_nytimes_json();
  println(" load random image by mouse click ");
}

void get_image() {
  int whatpic = int(random(master.size()));
  imageURL = nyURL+master.getJSONObject(whatpic).getString("url");
  if (dprint) println("whatpic: "+whatpic+" image link: "+imageURL);
  img = loadImage(imageURL);
}

void draw() {
  background(200,200,0);
  if ( img != null ) image(img, 0, 0);
}

void mousePressed() {
  get_image();
}
2 Likes
3

Thank you so much! The problem is solved. I made a String array and I appended URLs for different pages into it. Then I saved the array and used the file in another sketch to visualize what I had.