I'm looking into doing something similar as well, may I know how long have you been collecting the data, and how do you decide which data to collect?
I have been collecting data from the twitter API for a few days as well. I wanted to get an idea of average tweeting pattern, but without access to the firehose API, I got a feeling that the sample I have isn't very "neutral" as I have been mostly pulling from the popular and local tweets endpoints.
Any advice on how should I approach these kind of data collection?
If user streams are not releavant to you, you may use the `filter` endpoint instead of `sample`, and focus on keywords describing a relevant niche for your analysis.
In case you want to limit yourself to tweets geo-located in a certain location, you have to be aware that the bounding box filter of twitter is buggy: it will return you tweets geo-located outside of the desired area, and you are not sure to be getting all tweets.
I have been collecting data from the twitter API for a few days as well. I wanted to get an idea of average tweeting pattern, but without access to the firehose API, I got a feeling that the sample I have isn't very "neutral" as I have been mostly pulling from the popular and local tweets endpoints.
Any advice on how should I approach these kind of data collection?