Fix "cannot allocate memory error" for large XML dumps
We get a “cannot allocate memory” error when we try and dump large XML records from USASpending—-like on the order of 1000 records (or more, as in the Caltech grants example). Tried everything from increasing cache, increasing locks, etc. We actually get the data, that’s not the problem (and it’s only around 9MB), so I’m not sure why there’s such a problem putting it in the database. Maybe we have to put it in in smaller chunks? Very strange.
Comments and Changes
Part of this is due to our search, in that it pulls up much more information than I think we need. We need to figure out a way to pare done these search results.
We can do this using the zip code, but we need to do this carefully, as it will require many changes across all of our main files.
- Milestone changed from Credit Swaps (v 0.2) to REIT (v 0.3)
I’m still not sure what is going on here. I think it’s actually a bug in their system; i.e., even when I try and enter in the correct information, I’m not able to get DoD grants unless I use the recip_id, even if what I put in for “recipient_name” matches what “recip_id” points to. I’ve e-mailed the usaspending folks to see what’s going on. Moving this to the next milestone.
Feeds