r/rust•
zoells

Download source for all crates that depend on a specific crate?

I'm trying to evaluate some changes to a crate I help maintain, and would like to see how a portion of the public API is currently being used. At a 50,000 foot view, I really just want to clone the source for crates that depend on my crate directly, and then rg/grep over them.

Is there a tool that reports the repository URLs for crates that depend on some specific crate? I could probably crawl crates.io/libs.rs, but I'm hoping this specific use-case is already met.

Posted Jul 8, 2022, 12:59 AM UTC

67
[deleted]•

[deleted]

5
ehuss•

Here's a hacky way to get every reverse dependency:

  1. Clone the index: git clone --depth=1 https://github.com/rust-lang/crates.io-index/
  2. Remove some stuff: rm -rf crates.io-index/.git* crates.io-index/config.json
  3. Use jq to select every package that depends on your package, and output each as a URL to a file (replace "cargo" with the name of your package): find crates.io-index -type f -exec tail -1 {} \; \ | jq -r 'select(.deps | .[] | .name == "cargo") | "url=https://static.crates.io/crates/\(.name)/\(.name)-\(.vers).crate"' > urls
  4. Download all of those crates: curl --http2 --remote-name-all -K urls
  5. Extract all of those crates: find . -name '*.crate' -exec tar -xzf {} \;

The tail -1 selects only the most recently published version which is a bit primitive. You could use cat if you want every version. More advanced semver comparisons would probably require writing a script. This could be done in Rust with a fairly small number of lines. jq is also pretty slow, so a script would be much faster.

3
tcisme•

Thank you, generous bash warrior. (I had to remove a backslash from step 3.) I used this to grep to find examples of a crate's methods being used (maybe there is a better way?)

1
vlmutolo•

Is your crate on crates.io? If it is you can see other crates that depend on yours by navigating to your crate and clicking "dependents".

1
zoellsOP•

Yeah, I'm aware of this, I'd just prefer not to manually clone down each repo or navigate the website. It'd be reasonable to do once, but I'm imagining doing this multiple times in the future.

3
vlmutolo•

Yeah fair. I was thinking you could scrape that page with something like scraper.

There's a made-for-analysis crates.io dump updated daily at:

https://static.crates.io/db-dump.tar.gz

But it's 200MB compressed (1GB decompressed). Not sure how often you want to download that

3
spaun2002•

For example: https://crates.io/api/v1/crates/libc/reverse_dependencies But first read the crawling policy for the crates.io

3
bobbytabl3s•

A bit more complex but it would be cool to have a tool which automatically sends PRs to all affected libraries with appropriate code changes (when this can be automated).

1
alsuren•

Remember that not all usages of your crate will be published to crates.io. If you are expecting to have a lot of open source applications using your code, also do a GitHub search, using TOML as the language:

https://github.com/search?q=proptest-derive+language%3ATOML&type=code

There is also a repo of https://github.com/rust-lang/rust-repos that I have found useful for data mining in the past (guess the location of Cargo.toml and curl the raw URL for it for every repo). The GitHub search API is probably more appropriate for your purposes though.

1