Source
r/rust•
shadowmint

why Cargo.lock?

Can someone explain clearly to me what Cargo.lock is, and why it exists?

The guide (http://crates.io/faq.html#why-do-binaries-have-cargo.lock-in-version-control,-but-not-libraries?) says:

The purpose of a Cargo.lock is to describe the state of the world at the
time of a successful build. It is then used to provide deterministic 
builds across whatever machine is building the project by ensuring that 
the exact same dependencies are being compiled.

...but my cargo.toml has explicit versions in it:

 [dev-dependencies.fixture]
 git = "https://github.com/shadowmint/rust-fixture"
 tag = "0.1.2"

There's no confusion here.

Why do you need a separate file that just stores a copy of exactly the same thing?

The guide seems to suggest that Cargo.lock stores the exact version of a dependency, while the Cargo.toml might only store a semver restriction (eg. >= 1.0).

So you can get absolutely repeatable builds using exact versions of libraries.

Fair enough I guess... but if I want that behaviour, I can just pin exact versions in my Cargo.toml.

...the guide also goes on to say:

This is precisely because a library should not be deterministically recompiled for all users of the library.

Well. I feel the same about this as above; if the library author requires that a specific version of dependencies of a library be present, just pin them in cargo.toml.

If the parent project wants to pin the dependencies of an arbitrary child library, why not put those details into the cargo.toml of project? eg.

[lock-dependencies.xml]
git = "https://github.com/netvl/rust-xml"
version = "1.0.2" <--- Lock any reference to rust-xml that is not explicit to this version

Another alternative would be to use the npm model where each build is only as deterministic as the package.json requires it to be.

Cargo.lock just feels like it's been inherited from ruby's bundler for no reason.

Can someone explain the thoughts behind it?

I'm happy to have my opinion of it changed if there are some good reasons, but at the moment the only time I ever hear anything about cargo.lock its people saying "Yes, add it to your gitignore".

Posted Oct 9, 2014, 1:36 AM UTC

1324
polyfractalturbine•

Lockfiles are important for a variety of reasons: team collaboration, CI servers, debugging bugs on other systems, etc.

Usually you want to specify SemVer restrictions and not exact values, because you don't actually care about exact versions. You just want the latest stable version in the series you are working with. So you add ">=1.0, <2.0" to your Cargo.toml, update dependencies, build locally and it works great.

Push to Github, your CI server sees the git commit hook and performs a git pull, initiates a build and runs your test suite. Alas, there was an error! You can't seem to reproduce it locally, so you have to debug on the CI server itself (gross) and eventually realize it's a bug in one of your dependencies.

Why didn't you see this locally? Because in the time between your local build and the remote build, one of your dependencies updated and introduced a bug.

Lock files would have prevented this. Update dependencies locally whenever you feel like it. But remote builds (or other team members, whatever) can checkout the lock file and deterministically have the same deps as you.

There is nothing more maddening than trying to debug a problem which turns out to be mismatched dependencies.

12
shadowmintOP•

Ok, fair enough.

So why isn't there a 'cargo lock' command to generate a Cargo.lock if you want one?

Is there meaningful benefit as having these (typically ignored and discarded) files hanging around by default? (esp for libraries where the advice is to ignore and delete these files and not keep them)

Your remote build fails?

cargo lock
git commit -a -m "lock versions" && git push

Watch and see what happens.

(my feeling is that if this was an optional step, people wouldn't really use it much)

3
xstr•

It's there, but not mentioned neither in the default short help nor in the guide.

$ cargo --list
Installed Commands:
    bench
    build
    clean
    config-for-key
    config-list
    doc
    fetch
    generate-lockfile
    git-checkout
    help
    locate-project
    login
    new
    package
    pkgid
    read-manifest
    run
    test
    update
    upload
    verify-project
    version

$ cargo help generate-lockfile
Generate the lockfile for a project

Usage:
    cargo generate-lockfile [options]

Options:
    -h, --help              Print this message
    --manifest-path PATH    Path to the manifest to generate a lockfile for
    -v, --verbose           Use verbose output

(There is also a common puzzling error situation 「A Cargo.lock must exist before it is updated」 that I see people keep stumbling upon and wondering what to do next)

5
steveklabnik1rust•

That's because generate-lockfile isn't really supposed to be used, a lockfile represents a successful build. But we're in a weird moment with everyone on HEAD and not using real versions, so using it to fix questions like that one in IRC is okay, for now.

The correct command to generate a lockfile is cargo build.

1
polyfractalturbine•

What u/xstr said.

Is there meaningful benefit as having these (typically ignored and discarded) files hanging around by default? (esp for libraries where the advice is to ignore and delete these files and not keep them)

For you they may be "typically ignored and discarded" but for other people they are frequently used.

(my feeling is that if this was an optional step, people wouldn't really use it much)

Agreed. Which is why it would be a bad default :P

My opinion is that defaults (for any system) should be as safe as possible, and as introduce as few surprises as possible. This satisfies both. It provides the strongest deterministic guarantees and prevents strange bugs caused by mismatched dependencies.

If you don't want to use it...don't. Add it to your .gitignore (one command, ever) and forget about it. If the default was inverted, you would have to execute the cargo generate-lockfile command every time something weird happens and check in the new lock file (two commands, repeatedly executed), presumably after you've already built/committed/pushed/built somewhere else/found bug.

3
[deleted]•

[deleted]

1
vladley•

Should I add Cargo.lock to my VCS?

1
steveklabnik1rust•

Another thing that people are missing in this thread: a lockfile lets Cargo not do the whole dependency resolution step upon every build. It can just use the information in the lockfile instead. This is much, much faster.

5
emk•

Having worked on lots production systems that use Ruby, Gemfile.lock is by far my favorite Bundler feature. Let me use my heroku-rust-cargo-hello as an example. This project supports guaranteed one-click deployment to Heroku by novice Rust users, despite the ongoing churn in Rust.

What's in Cargo.toml

Officially, we have only three dependencies:

[dependencies.iron]
git = "https://github.com/iron/iron.git"

[dependencies.router]
git = "https://github.com/iron/router.git"

[dependencies.logger]
git = "https://github.com/iron/logger.git"

In the future, each of these will be a simple SemVer depdendency: "We depend on anything that's backwards-compatible with version 1.1.4 of iron." Occasionally, if I discover that, say, logger 1.2 accidentally breaks compatibility with version 1.1.8, I'll restrict the official dependency a bit further, as a way of communicating, "SemVer failed this time." But through all this, my Cargo.toml remains nice and clean.

When I'm actively developing an application, I can easily say "Give me the latest version of all my dependencies," or "Give me the latest version of libfoo", or "OK, update everything except libbar to the newest version, but leave that bound to something >= 1.3.7 but < 1.4, because libbar 1.4 accidentally breaks libquux for various tedious reasons." And at no point in this process do I need to manually maintain the exact versions of every library that I depend on directly or indirectly. (I need to do this with production projects using Bower, and manually updating all those version numbers is a painful process.)

What's in Cargo.lock

I won't paste the whole thing here, because it's long and hideous. But indirectly, my application currently relies on 16 different libraries, many of which I know nothing about. And for each of these direct and indirect dependencies, Cargo.lock records which versions (1) are known to actually work, and (2) have undergone testing. For my real production projects (none of which are in Rust yet), this testing may represent extensive work by an entire team of people.

This is especially useful if I release FooApp 1.6 into production, move on to other things, and get called back to work on a bug 9 months later. Using the *.lock file allows me to exactly reconstruct the production version, fix a bug, and release/deploy it without accidentally upgrading 37 dependencies at the same time, which is obviously Not a Best Practice. And because Cargo.lock is generated automatically, I won't discover that somebody forgot to generate it, leaving me totally hosed.

Rules for tracking version dependencies and staying sane

  1. Libraries should only have loose, SemVer restrictions on their immediate dependencies. Anything more specific makes it impractical to combine libraries. In other words, do not commit Cargo.lock for libraries! (Cargo knows this, and actually adds the lockfile to .gitignore when generating a library.)
  2. When developing applications, it's important to maintain a short, clean list of immediate SemVer dependencies, plus a few special cases where SemVer has broken down.
  3. When deploying or releasing applications, it's important to always have an exact list of what went into the tested, working configuration. This should not rely on every single developer remembering to update a huge list of indirect dependencies, because that will never happen reliably in practice.

Basically, Cargo.lock exists to make sure we satisfy both (2) and (3) in a very agreeable way: We have one list of human-maintained dependencies, which is short, clean and readable. And we have a second list of dependencies which exhaustive and exact, and which does not rely on human discipline to create.

I've spent years working with Bundler and Bundler.lock for many different consulting clients, and I'm extremely satisfied with the system. I've also worked extensively with npm's package.json and npm-shrinkwrap.json, which are a clunkier implementation of the same idea, but which are still usable as long as people always remember to run npm shrinkwrap at appropriate times. And I've worked with Bower, which doesn't understand locks, and which requires careful manual tracking of indirect dependencies, and which still results in lots of grumpy emails from a release team when one of the 15-odd JavaScript dependencies messes up SemVer.

Anyway, I hope this explains why Cargo works the way it does, why we might want a split between official dependencies, and "what's definitely known to work," and why it's helpful that Cargo.lock is maintained automatically without anybody needing to run a manual command.

4
shadowmintOP•

So, in a nutshell, we have two models here, used widely in production:

  1. Ruby with bundler, where this works the same as cargo

  2. Node with npm, where it relies on the developer doing the right thing

...and basically in cargo we have a situation "... which does not rely on human discipline to create."

We've decided to go down the bundler route instead of the npm route, because we don't trust our developers to Do The Right Thing.

I question that decision personally, but fair enough.

Thanks for the comprehensive answer~

1
dbaupprust•

we don't trust our developers to Do The Right Thing.

Do you get everything perfect always? It's unreasonable to expect things to always go right. It's a major reason for Rust to exist: humans are human, and being guaranteed to Do The Right Thing without a machine checking (i.e. writing C/C++) just does not work.

6
polyfractalturbine•

we don't trust our developers to Do The Right Thing.

There are so many important things that I do need to get right when developing, the last thing I want to spend time thinking about is how to carefully manage dependencies in a collaborative team environment with CI servers, etc. I'll happily let software manage the problem for me

Besides, a lock file is such a harmless "burden"...it only helps. I've never once had to fight with a lock file because it won't let me do what I want. If you don't want it, fine, but defaults should be safe.

As an aside, PHP's composer does similar to Ruby's Bundler.

2
emk•

At least in my experience, there are very few downsides to the Bundler model. I almost never think about Gemfile.lock. I commit it for applications and ignore it for libraries. I run bundle update as needed, and occasionally tweak a constraint in my Gemfile. In other words, we get a very nice workflow with a lot of benefits, and we don't seem to pay anything to get it.

Once Rust becomes stable, and Cargo supports a package repository with version numbers, everything should come together very nicely.

1
solar-ice•

Node with npm, where it relies on the developer doing the right thing

Sort of, ish. Any project that makes it anywhere close to production tends to have an "npm shrinkwrap", which is similar to these lock files, or otherwise all the dependencies are checked into the repo.

1
mcpherrinmrust•

You can't just pin the version in Cargo.toml:

Say you use a library foo, and that library has a semver restriction on its dependencies. You would have to fork all those dependencies to pin their Cargo.toml versions down.

Also, it pins the exact git checkout, which is good if the library dep is just version 0.0.0 and you're just updating once in a while, making sure it works. I haven't verified, but I think this also prevents an upstream library from changing itself underneath you maliciously.

2
mcpherrinmrust•

And to continue: Not to mention it would just be straight-up annoying to have to go edit all your semver restrictions to limit them to what cargo has resolved: It did all that hard work for you, so why not use it?

During development you can probably ignore it, but I would check in the .lock file when tagging an application's released version so you can later recreate the exact same code, even if new library versions have been released and the semver resolver would come to a new conclusion.

2
shadowmintOP•

I think the git# is a bit of a strawman argument given that ultimately cargo will use version numbers not git hashes (and won't be intimately related to git or any other version control system, which may or may not expose any version information at a finer grain level than semver).

I guess it's useful(?) to pin the exact version of all the dependencies automatically in cargo.lock, if you assume that no old versions of libraries ever drop their tags (which is a major pain point in npm builds). hmm...

1
[deleted]•

[deleted]

1
loganekz•

This to me would be the only compelling reason why you would need to use a lock file.

But quite frankly can be avoided all together if you simply require transitive dependencies to specify an exact version. I don't know any sane library developer who would not do this in the first place anyway.

1
fgilcher•

Consider the following case:

  • Library A is maintained by Party P
  • Library B is maintained by Party Q and depends on A, version 1.1.x
  • Library C is maintained by Party R and depends on A, version 1.1.x as well

Library A releases 1.1.x+1. A contains a non-API-breaking change for a problem that P has. Specifiying the exact version of A in B and C means that P has to wait for Q and R to both decide to move to the new version of A and release their new version before upgrading. It also requires that both switch, maybe C is unaffected by the problem an decides to stay one patch version behind. Suddenly, A cannot build because of conflicting versions, although this should not be a problem.

This all could be avoided if we just let Cargo actually resolve those dependencies and let the library authors define lenient expectations.

5
loganekz•

I understand that some might consider this a bad trade off but I would consider ensuring library authors pin versions of any transitive dependencies they would impose on their users a good thing.

-1
fgilcher•

I understand that some might consider this a bad trade off but I would consider ensuring library authors pin versions of any transitive dependencies they would impose on their users a good thing.

For the reasons described, this only holds true if they are all by one vendor.

Imagine the aforementioned patch is a security patch.

Also, cargo does allow to fix the library version (by just expressing a strict dependency), which is exactly what you want if you deliver a product as multiple libraries. That is, for example, what Rails does. Rails is a actually a set of libraries with very strict dependencies internally.

So Cargo allows to do both and to insure repeatability, writes down the result of that operation.

2
riccierirust•

This becomes a pain as soon as there is more than one library (or one library and the main application) depending on the same library. If libA depends on libFoo 1.1.0, and libB depends on libFoo 1.1.1, I will end up with two semver-compatible versions of the same library in my executable

2
loganekz•

In this case it needs handled by cargo anyway and I'm assuming it is resolved in some consistent fashion to generate the cargo.lock file. If this is true it seems it would be feasible to use the same rules without generating a lock file.

2
riccierirust•

AFAIK it is: both versions of the library are included on the final executable (or at least that's how I read these situations would be handled). That's consistent, but far from ideal. If both libraries had expressed their dependencies as semver this wouldn't happen

1
steveklabnik1rust•

While it's on Bundler, this article is still true for Cargo: http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/

1