On concerns about Haskell's Prelude favoring Foldable/Traversable
yesodweb.com/blog/2014/10/classy-base-prelude
yesodweb.comPosted Oct 6, 2014, 3:40 AM UTC
64101
yesodweb.comI just want to remind everone, that if you're on Debian or Ubuntu, you can easily install nightly GHC HEAD bindists and try how the BBP/FTP/AMP-lified Prelude feels like, and see for yourself where it causes actual problems:
PS: and don't panic if GHCi starts up less verbosely, as we've changed the verbosity for the RTS linker chatter, so GHCi starts up tersely as
GHCi, version 7.9.20141003: http://www.haskell.org/ghc/ :? for help
λ:2> Control.DeepSeq.rnf ()
()
it :: ()
λ:3>
as opposed to
GHCi, version 7.8.3: http://www.haskell.org/ghc/ :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
λ:2> Control.DeepSeq.rnf ()
Loading package array-0.5.0.0 ... linking ... done.
Loading package deepseq-1.3.0.2 ... linking ... done.
()
it :: ()
λ:3>
PS2: and in case anyone wonders about the prompt, I've got
:set +t
:set prompt "\955:%l> "
:set prompt2 "\955:%l| "
set in my ~/.ghci
20
:set +t :set prompt "\955:%l> " :set prompt2 "\955:%l| "
nice
5
You can just use 'λ' instead of '\955' FWIW.
1
I know... but I had to avoid UTF8 encoding for some reason I don't remember anymore (maybe some locale issue I had with older GHCs), so I stuck w/ \955 which is plain ASCII.
1
The page for Foldable/Traversable would explain
1.) what is being changed 2.) which changes create a breakage 3.) how type errors have changed 4.) how library code is affected 5.) how user code is affected 6.) best practices for using Foldable/Traversable
We have committed to post a page on the state of the Foldable/Traversable proposal, including the parts where we feel we need direction from the community about how best to proceed.
This stalled somewhat when hvr pointed out the scope of the breakage caused by AMP in #9590.
Unfortunately, the timing of things wound up a bit off.
Austin posted the release notes for what was happening in GHC and Neil's first post on this topic went up while I was stuck at IFL 2014, and concerns rightfully raised by folks in the first 2 threads Neil started on this topic further changed what it was that I had intended to say in the writeup.
As a result i've been flailing around here writing up piecemeal replies to folks rather than putting together the coherent document I'd originally promised to Herbert. =/
I'll pull together a frank summary of the current state of things and open design options, and importantly how and why they entangle with issues for AMP and Lennart's candidate MonadFail proposal shortly.
Right now we are stuck in a loop of repeating the same points that were already made in the original discussion of the proposal. Given a wiki page, Neil and others could point out the down-sides of the proposal with actual code and have their voice heard in a productive way that builds up our body of knowledge.
Hear, hear.
11
Usually in Python they write the proposal before committing the code into the repo.
1
We have a proposal.
It is the Foldable/Traversable Proposal / Burning Bridges Proposal a year and a half ago. It garnered over 100+ comments, across 2-3 threads, heavily biased in the positive on the topic of Foldable/Traversable generalization.
A large part of the reason for the formation of the committee was to manage the sense of frustration that folks in the community had that nothing could ever change without complete universal agreement on any package that didn't have a dedicated maintainer. As more and more of the platform fell nominally into GHC HQ's hands there was nobody there who felt responsible for making decisions.
Mind you, the proposal itself just said "swap out the monomorphic versions of combinators in base with the ones from Foldable and Traversable" and after SPJ formed the committee, and once we collectively figured out how to work with GHC HQ we did.
That part is done.
It was easy except for a few knock-on effects that we found when we went to implement it. Data.List and Control.Monad re-exported monomorphic functions, and we had to restructure parts of base to put the code in the right place to allow the re-exports. Finally, since we wanted to do it without changing semantics in user code, we needed the raw material from the "extra members in the Foldable class" proposal from a few weeks back.
If we just want that proposal, it is summarized in that sentence directly, and already implemented.
Anything around haskell2010 or trying to make things better for folks who would prefer a more monomorphic Prelude is actually an extension to the scope.
If you want us to go farther and try to make it easier for folks to try to work with a monomorphic Prelude, these are the things we need help with, but they are technically out of scope of the original proposal.
I personally want that smoother upgrade path, but we could live without it.
14
It is the Foldable/Traversable Proposal / Burning Bridges Proposal a year and a half ago. It garnered over 100+ comments, across 2-3 threads, heavily biased in the positive on the topic of Foldable/Traversable generalization.
Where was it posted? Was it easy to miss?
2
It was a proposal on the libraries@ mailing list under "Burning Bridges", "Foldable/flexible bridges", and several other titles that raged for a couple of months, and more or less overwhelmed the mailing list in May 2013.
The formation of the core libraries committee spun out of the "Making decisions" thread that SPJ formed in response to the furor.
7
The proposal "Swap out the monomorphic instances in base with the ones from Foldable and Traversable" doesn't say anything about exporting new names from Prelude like foldMap, mappend, and traverse. I don't see how it is logically necessitated by generalizing the functions which are already exported by Prelude. Someone who actually wants to write a Foldable or Traversable instance can import the necessary modules.
Was this simply a mistake? If not it looks exactly like committing the code into the repo before writing the proposal.
1
I would argue that it was a decision.
We have a Libraries Submission process.
The maintainer is trusted to decide what changes to make to the package, and when. They are strongly encouraged to follow the guidance below, but the general principle is: the community offers opinions, but the maintainers decide.
The core libraries committee acts as a collective maintainer for the portions of the Haskell Platform that aren't maintained by anyone else.
We had a nuance to the proposal that required a decision, and so we made one.
Admittedly we also have,
API changes should be discussed on the libraries mailing list prior to making the change, even if the maintainer is the proposer. The maintainer still has ultimate say in what changes are made, but the community should have the opportunity to comment on changes. However, unanimity (or even a majority) is not required.
Changes that simply widen the API by adding new functions are a bit of a grey area. It's better to consult the community, because there may be useful feedback about (say) the order of arguments, or the name of the function, or whatnot. On the other hand few clients will actually break if you add a new function to the API. Use your judgment.
The precise space of names we want to add to the Prelude is still open for discussion. It is in the space of things we need to talk about -- we have as-yet unmerged changes into GHC around this question, but I do personally think the balancing act of utility vs. complete non-breakage is best served by making the classes that are exported from the Prelude implementable as given to you.
6
Wait, the other names are going into Prelude? Why? We don't export all of Control.Monad from Prelude, why should traverse be any different?
4
This article is well written and persuasive - nice! There are two concerns:
- The code was not proposed, only a guiding design principle. That seems dangerous for something so critical. You may have insights that show their interpretation of the design is flawed. You have actually done the work that I am concerned has been skipped, so I want your opinion!
- The plan for haskell2010 to contain the previous Prelude and tell beginners to use that is the current plan. It seems few think it is a good idea, and I don't think it will even work.
5
The plan for haskell2010 to contain the previous Prelude and tell beginners to use that is the current plan. It seems few think it is a good idea, and I don't think it will even work.
It was more that if we say that a more monomorphic teaching Prelude is a thing we want to offer, haskell2010 is already in place so if possible it'd have been nice to be able to employ it for its stated purpose.
I agree that the haskell2010 plan does not appear salvageable at this point.
The AMP looks to have killed it.
haskell2010
Given the tools we have in the bag today, at best that package can live in limbo, near to 2010 with either...
1.) Applicative bolted in, pushing it further away from the language standard
2.) its own Monad, rendering code written with it incompatible with hackage.
3.) or something like a horrid infectious overlapping, incoherent instance Monad m => Applicative m, which also effectively renders it incompatible with Hackage.
I was holding out hope that at Hac Phi in a couple of weeks I could work with Richard Eisenberg on his version of InstanceTemplates, and see if we could prepare it in time for 7.10.
If we had that in place we could put in a default superclass default instance for Applicative in terms of Monad, which would resolve a number of issues. In particular there is a largish portion of hackage that hasn't acted on the Applicative-Monad Proposal warnings, so it'd enable us not to have the option to not break their code, but it'd also enable the haskell2010 package users to still be able to define new monads without having to pull us any further away from the standard if we put a default definition for Applicative in in terms of the Monad.
If we're forced to make a decision with the tools we have today then that fourth option ceases to be an option.
Prelude alternatives
If we abandon that package, which we basically have done, anther option seems to be to go with something closer to your original option of a second Prelude, possibly inverted, with an explicit Prelude.New (or Prelude.Old depending on selected direction), alongside with a passing proposal made by Simon Marlow a year and a half ago that if you import Prelude.Foo then NoImplicitPrelude would get set automatically. This would make alternate preludes easier for folks to push.
An argument for it being Prelude.New with the new behavior deferred for a year as Yitzchak Gale suggested is that we wind up with Prelude already somewhat deformed from AMP, so it isn't really the 'Old' Prelude anyways.
An argument for having the monomorphized Prelude.Old be the second class citizen is that things are set up to avoid systemic breakage and that the candidate user base is basically a subset of the user base for haskell2010, which has seen very little adoption.
However, there is a huge caveat with both of these... they can't be implemented as such.
Why?
Control.Monad re-exports the monomorphized mapM, mapM_, etc.
Control.Monad.Foo from mtl repeats this sin.
Data.List re-exports monomorphized versions of everything, and is commonly imported unqualified as it exports the same version of the combinators from Prelude.
etc.
These are the very things that make importing Data.Foldable and Data.Traversable such a chore for users today. They have to hide the existing combinators and then keep on hiding them!
Prelude isn't just one module being affected, but rather a large chunk of base (and mtl) that redundantly exports the same combinators is also affected.
We could explore this option further if we also then go back and remove these re-exports -- rather than simply reform them to match the other types as we'd been planning. However, that increases the scope of breakage for 7.10 over what is strictly necessary to implement the raw original proposal.
[Edit: I changed the link for Richard's proposal from DefaultSuperclassInstances to InstanceTemplates to get the right one]
14
In particular there is a largish portion of hackage that hasn't acted on the Applicative-Monad Proposal warnings
I have the theory that cabal install foobar quieting down the build-process may have had an impact here, as the warnings would not be visible anymore unless you explicitly use -j1. Otherwise I guess people would have started nagging the respective maintainers to fixup the AMP warnings more actively by now.
Next time we'll simply add a threadDelay 60000000 right after such a warning is emitted. That should catch everyone's attention (just kidding... obviosly) :-)
5
proposal made by Simon Marlow a year and a half ago that if you import Prelude.Foo then NoImplicitPrelude would get set automatically. This would make alternate preludes easier for folks to push.
That is a really nice idea.
5
Even though it is a breaking change for a few modules of mine, and feels a bit ad hoc, it is simple enough to explain to folks.
3
It's true that it's ad hoc, but after all, Prelude is already an ad hoc magical name, so it fits. Sorry about your breakage though.
5
I was holding out hope that at Hac Phi in a couple of weeks I could work with Richard Eisenberg on his version of DefaultSuperclassInstances, and see if we could prepare it in time for 7.10.
If we had that in place we could put in a default superclass default instance for Applicative in terms of Monad, which would resolve a number of issues. In particular there is a largish portion of hackage that hasn't acted on the Applicative-Monad Proposal warnings, so it'd enable us not to have to break their code, but it'd also enable the haskell2010 package users to still be able to define new monads without having to pull us any further away from the standard.
what is the status of that proposal anyhow? I gathered the https://ghc.haskell.org/trac/ghc/wiki/IntrinsicSuperclasses and/or possibly https://ghc.haskell.org/trac/ghc/wiki/InstanceTemplates superseded that particular design, based on the note on the top of the article?
So is Intrinsic Superclasses the current contender? I've hoped for something like that for ages, and that design seems reasonable, possibly modulo some syntax bikeshedding...
3
IntrinsicSuperclasses is Conor McBride's preferred version of things.
DefaultSuperclassInstances [edit: actually, InstanceTemplates] is Richard Eisenberg's.
Ultimately I think both of them contain good ideas and they could probably stand to cross-pollinate better.
Trying to see if there is a viable timeline here is a large part of why I'm going to Hac Phi this year.
7
DefaultSuperclassInstances is Richard Eisenberg's.
hm are you sure about this? goldfire created the InstanceTemplates trac page, while pigworker created both IntrinsicSuperclasses and DefaultSuperclassInstances pages ?
3
Why don't you think it will work?
The only obstacle is the Applicative Monad proposal. The worst case scenario there is "infecting" haskell2010 with Applicative. It has no impact on keeping currently monomorphically list functions in the prelude as monomorphically list functions.
5
Using the haskell2010 package and the base package in the same program doesn't really work, since all the module names clash. Not being able to access base cuts out a lot of things you need to do. Therefore, using the haskell2010 package is a terrible idea. In contrast the haskell98 package used to only use non-hierarchical names and so could be used with base, but then they added Prelude to haskell98, and broke it.
I don't care about adding Applicative to haskell2010. The package itself doesn't work, so whatever you do to it is irrelevant. But that does mean you can't rely on it to become "the beginner" Prelude.
2
I fail to understand this. I thought you wanted it as a teaching package, not as a package for students to continue to use. What is there in base but not haskell2010 that you feel is necessary for teaching?
5
I don't want a teaching package, I think a teaching package is a terrible idea. I want a package for general use. I think teaching something and then having a cut off point 4 months in where you say "you're a big boy now, you get to play with real Haskell" would alienate everyone (and I can't imagine any teacher doing that, because they'll skip the teaching package).
4
This article is well written and persuasive - nice!
That means a lot coming from the person I was quoting, I am glad that the debate is not creating animosity.
-
It would be good to solicit input from authors & users of prelude replacements that have taken a similar approach, but right now I don't know the details of what is being changed.
-
It doesn't make sense to me to have a Haskell standard that is inoperable with current code. Am I wrong that the starting point for this approach would be a 2015 language revision with just the changes needed to make the standard work (Applicative/Monad)? Does it then make more sense to create a Prelude replacement for these purposes ahead of creating a new standard?
-
I don't know if the plan is to tell beginners to use something else so much as the plan is just to have that option available.
-
I think a more promising solution to the problem of generic type complexity is making specialization of type signatures easier in code, documentation, and compiler error messages. Unfortunately these are fairly new concepts and there is not a lot of time before the 7.10 release is supposed to be made.
4
Standard process seems so dead, could even just the Applicative/Monad haskell2015 actually happen? It certainly should. Regardless, I don't think Applicative/Monad should wait for the standard any longer, so many years have passed for that wart to finally disappear. And now its in HEAD!
2
I said persuasive, not that it persuaded me :) - but it's the kind of thing that might. Certainly no animosity on my part! I also note that you made 38 minor tweaks to the Prelude, and that once bundled with GHC, that kind of thing becomes massively harder.
- Wholeheartedly agree! We need to know what we are getting into, and then people who have travelled this road need to comment on it. You have 38 releases worth of minor tweaks, I don't want us to reach feature parity in GHC 7.86.
- Not sure that's necessary, the code behind the standards isn't as important, since the haskell2010 package is basically unusable.
- Beginners need clear and unambiguous guidance so they can focus on the language. Having a beginner Prelude is a bad idea. Having a beginner Prelude they aren't expected to use is an even worse idea.
- Yep, that's part of why I wish things were moving slower. Get feedback. Make improvements. Do better.
1
You neglect that classy-prelude is in fact an ambitious change of the prelude. The changes under discussion here are relatively minor and involve the generalization of a few functions.
8
This article is well written and persuasive
While indeed well written, well intended, and appreciated, I don't find this article persuasive at all about Foldable and Traversable.
I don't agree that namespacing and qualified names are the issue here. To me, the issue is: should it be easier to write code in a generic style, or in a style with the most precise denotational semantics. As an application developer, I see the latter as Haskell's biggest strength, and that is what I would much rather have.
Greg is a major contributor to the wonderful Yesod ecosystem - of which we are a major user. As a framework which needs to be very general, and especially as one that was originally inspired by many of the features of dynamic web frameworks like Ruby on Rails, Yesod does require a lot of very generic code. But just as Greg argued eloquently against favoring beginners over programmers, the same applies to not favoring generic library authors over the application developers for whom the libraries are being written.
Lists are not just simple - they are a fundamental semantic concept. For almost every function you'll ever write, the most natural and semantically clear way to write it will involve lists. Occasionally, when you need to optimize and when the compiler's implementation of some particular list happens to be your bottleneck, you do need to switch to some other data structure. And I understand that for Greg, who needs to support all of those variations, list-by-default will indeed cause him to need to type extra characters quite often, or to train his IDE to do that for him. But for the case of general application programming, list-by-default has proven itself to be the right choice.
EDIT: the same goes for Edward and his absolutely amazing generic libraries, such as his lens library, parser library, abstract algebra libraries, and category theory libraries. And thanks for all the great work on this proposal. As I have said - my mind is open, I'm willing to give it a try.
Improving the library process - yes, great suggestions, I completely agree.
1
Are typeclasses with laws really mutually exclusive to precise denotational semantics?
6
No. But they make the semantics depend on other code that can be very difficult to find. What is the type of that function? Is it a method of a class? If so, where is that class defined, and where is the instance defined? You can't necessarily find that out very easily from just the imports in the current module, or possibly not even with the imports together with the build-depends: field in the cabal file.
Even parametric polymorphism has a cost to semantic clarity. What are the types of the arguments of this function?
So by clear semantics I mean not just that precise semantics exist, but also that the semantics of each line of code are immediately apparent to a human reading just that line of code.
Not that I am against polymorphism - it is wonderful when needed. But when used gratuitously, it has a significant cost in clarity of code. I would much rather need to rely on my IDE to type a few extra characters here and there than to need to rely on it to understand the meaning of code.
1
I appreciate your appreciation of Yesod, etc. However, you have completely mis-characterized how I program libraries. There is very minimal usage of Foldable or Traversable in yesod, persistent, or any other of my libraries that I contribute to. Everything except for mono-traversable and classy-prelude defaults to using minimal abstractions, and almost all of the interfaces specify using lists. I only get to use mono-traversable and classy-prelude in my application code.
4
There's already a Trac ticket that mentions there's some more detailed design documents in the works - hopefully they end up on the wiki before too long.
4
Make sure if you pull some functions into the class and do this, you pull EVERY function into the class. We will lose a lot of performance doing this initially, and probably list fusion too. Maintainers need ways to override and buy into algorithmic complexity.
1
List fusion was a concern going in. It surprisingly hasn't manifested in practice as a benchmarking problem.
The extra components about what to fuse into the class was described in a separate thread on libraries@ a couple of weeks ago. The thread for that didn't advocate putting in "every function" but only just enough that the things that can be reasonably cached / calculated could be handled by individual containers as they saw fit.
5
I doubt anyone will see this now, but I'd like to put in a good word for qualified imports. They're common in Python, mandatory in Go, and they make import management much less fussy while still making the provenance obvious.
1
Nobody's keeping you from using qualified imports (if you want them). The not so nice thing would be to force everyone to use them instead of giving them a choice to opt-in.
2
From the original post:
...we still end up with needing to use module qualified names. This isn't really an acceptable way to program. I have not seen another language where this extra line noise is considered good style.
I was surprised to find this, since qualified imports are common and well-accepted in other languages. There's definitely some advantages to having things unqualified, but I don't think the qualified version is as unworkable as the article suggests.
1
Even when programming in Python, I prefer to import * from Foobar but Python makes this harder to the point forcing me to use qualified imports, as symbol clashes are reported eagerly rather than when actual ambiguities arise (as it's done in Haskell)
Moreover, while qualified imports in Haskell may be tolerable to some degree (but I'd rather avoid them), they become rather ugly/confusing when you need to disambiguate infix operators.
2
Yep, Python's guides consider the * to be bad style:
https://docs.python.org/2/howto/doanddont.html#at-module-level
The problem with import-all is that it assumes that name conflicts are rare between different modules. This can work out when the libraries have been designed to work together -- like the stdlib, or the scientific-computing toolkit in Python -- but it breaks down when you have a rich and varied ecosystem like we have in Hackage right now.
Of course, it's certainly very convenient when things do work out, enough so that it's worth taking a little time to avoid egregious conflicts -- particularly so for operators, since they're especially hideous when qualified. After a point, though, I do feel that namespacing is the least of evils.
1
RE last point:
Yes yes yes yes. I really think this is needed more that I noodle about it. I believe it has helped immensely with GHC and I think it's crucial for several reasons to help keep things moving.
Look at the Python Enhancement Proposals (PEP) index: http://legacy.python.org/dev/peps/
These are beautiful, straightforward, have a relatively standard format, are indexed for historical records, are revision controlled, and go through a standardization process like ours. Some are big, some are small.
As it stands, we have a 'process' (the last part), but it falls flat in several ways. We frequently hit points like:
A mailing list just doesn't scale for recording these particular things in an accessible way, honestly.
[email protected]is great for discussion and open forums, when we want to have an open discussion about what might happen, and let people voice concerns. But it is awful as a historical index/catalogue, and an awful place to get an overview - many times the discussion happens first, but rarely do refined abstracts of a proposal appear. Reading the full mail thread is almost always required to get a 'full, updated understanding' of what state something might be in, just as you said.I think adopting a process like the PEPs - with the primary motivation being a better way to document and evolve proposals - would be a great idea.
We do have a PEP process for the Haskell platform, modeled after the Python PEP process:
I must say it hasn't been a major success however.
It kind of sounds like no one knows about it. Could that be why it hasn't been successful or do you think there is something wrong with the proposal idea itself?
I think it's most likely a combination of issues, one of them is that it's not widely known.
Well, the way that Python handles that is to remind people of the PEP process when ideas that should have a PEP are proposed on the list without one.
I would love a PEP process. It also gives new users a way to see how the language has evolved, which can be very informative to get a good understanding of how to use the language.
If we need a quirky name we can just go with the acronym HIP/HOP = Haskell Improvement / Optimization Proposal / Project.
YAAAAAAS
HEP (Haskell Enhancement Proposal) even rolls off the tongue nicely.
... with a similarly viral reference/pun as PEP :D
...post-exposure prophylactic?
And it shows we're hep cats.
this sounds like a (further) formalization of the libraries list with even more process around it...
At some point (when enough avoid-success-at-all-costs-failure has been accumulated) adopting a more formal PEP-ish process will be unavoidable.
The fact these discussions have been so alive the past few days indicates that maybe we already have reached that point, and it's time. :)