> The reason is that they are not really a part of the language, they violate the spirit of the language, and in high-usage parts of code they quickly become a complete mess.
I'll admit that this is what I hate Python, and it's probably this spirit of the language as you call it. I never really know what parameters a function takes. Library documentation often shows a few use cases, but doesn't really provide a reference; so I end up having to dig into the source code to figure it out on my own. Untyped and undocumented kwargs? Everywhere. I don't understand how someone could embrace so much flexibility that it becomes entirely undiscoverable for anyone but maintainers.
Python type hints manage to largely preserve the flexibility while seriously increasing confidence in the correctness, and lack of crashing corner cases, of each component. There's really no good case against them at this point outside of one-off scripts. (And even there, I'd consider it good practice.)
As a side bonus, lack of familiarity with Python type hints is a clear no-hire signal, which saves a lot of time.
What I've experienced is that other factors make the biggest difference. Teams that write good tests, have good testing environments, good code review processes, good automation, etc tend to have fewer defects and higher velocity. Choice of programming language makes little to no difference.
Never has been an issue in practice...
I work at big tech and the number of bad deploys and reverts I've seen go out due to getting types wrong is in the hundreds. Increased type safety would catch 99% of the reverts I've seen.
Well, you could say that the problem in this case was the lack of documentation, if you wanted. The type signature could be part of the documentation, from this point of view.
Let me give a kind-of-concrete example: one year I was working through a fast.ai course. They have a Python layer above the raw ML stuff. At the time, the library documentation was mediocre: the code worked, there were examples, and the course explained what was covered in the course. There were no type hints. It's free (gratis), I'm not complaining. However, once I tried making my own things, I constantly ran into questions about "can this function do X" and it was really hard to figure out whether my earlier code was wrong or whether the function was never intended to work with the X situation. In my case, type hints would have cleared up most of the problems.
If the code base expects flexibility, trusting documentation is the last thing you'd want to do. I know some people live and die by the documentation, but that's just a bad idea when duck typing or composition is heavily used for instance, and documentation should be very minimal in the first place.
When a function takes a myriad of potential input, "can this function do X" is an answer you get by reading the function or the tests, not the prose on how it was intended 10 years ago or how some other random dev thinks it works.
So does Python:
That era of Python codebases were miserable to work in, and often ended up in the poorly though out "we don't know how this works and it has too many bugs, let's just rewrite it" category.
My position is that what is intended must be made clear between type hints and the docstring. Skipping this makes for difficult to read code and has no place in a professional setting in any non-trivial codebase.
This doesn't require type hints to achieve. :param and :rtype in the docstring are fine if type hints aren't present, or for complex cases, plain English in the docstring is usually better.
Proper type hints are typically very easy to add if the codebase is not a mess that passes things around far and wide with no validation. If it is, the problem is not with the type hints.
Type annotations can seem pointless indeed if you are unwilling to learn how to use them properly. Using a giant union to type your (generic) function is indeed silly, you just have to make that function generic as explained in another comment or I guess remove the type hints
That in itself violates the spirit of the language, IMO. “There should be one obvious way to do it”.
- There is one obvious way to provide type hints for your code, it’s to use the typing module provided by the language which also provides syntax support for it.
- You don’t have to use it because not all code has to be typed
- You can use formatted strings, but you don’t have to
- You can use comprehensions but you don’t have to
- You can use async io, but you don’t have to. But it’s the one obvious way to do it in python
The obvious way to annotate a generic function isn’t with a giant Union, it’s with duck typing using a Protocol + TypeVar. Once you known that, the obvious way is… pretty obvious.
The obvious way not be bothered with type hints because you don’t like them is not to use them!
Python is full of optional stuff, dataclasses, named tuples, meta programming, multiple ancestor inheritance. You dont have to use these features, but there are only one way to use them
Optional nature of those features conflicts with this statement. As optionality means two ways already.
But times change and these days, Python is a much larger language with a bigger community, and there is a lot more cross-pollination between languages as basic philosophical differences between the most popular languages steadily erode until they all do pretty much the same things, just with different syntax.
It never was a thing in Python, it is a misquote of the Zen of Python that apparently became popular as a reaction against the TMTOWTDI motto of the Perl community.
The misquote shifts the emphasis to uniqueness rather than having an obvious way to accomplish goals, and is probably a result of people disliking the “There is more than one way to do it” adage of Perl (and embraced by the Ruby community) looking to the Zen to find a banner for their opposing camp.
So on the language level it doesn’t directly change the behavior, but it is possible to use the types to affect the way code works, which is unintuitive. I think it was a bad decision to allow this, and Python should have opted for a TypeScript style approach.
Lots of very useful tooling such as dataclasses and framework like FastAPI rely on this and you're opinion is that it's a bad thing why?
In typescript the absence of type annotations reflection at runtime make it harder to implement things that people obviously want, example, interop between typescript and zod schemas. Zod resorts instead to have to hook in ts compiler to do these things.
I'm honestly not convinced Typescript is better in that particular area. What python has opted for is to add first class support for type annotations in the language (which Javascript might end up doing as well, there are proposals for this, but without the metadata at runtime). Having this metadata at runtime makes it possible to implement things like validation at runtime rather than having to write your types in two systems with or without codegen (if Python would have to resort to codegen to do this, like its necessary in typescript, I would personally find this less pythonic).
I think on the contrary it allows for building intuitive abstractions where typescript makes them harder to build?
Python’s types are machine-checkable constraints on the behavior of your code.. Failing the type checker isn’t fatal, it just means you couldn’t express what you were doing in terms it could understand. Although this might mean you need to reconsider your decisions, it could just as well mean you’re doing something perfectly legitimate and the type checker doesn’t understand it. Poke a hole in the type checker using Any and go on with your day. To your example, there are several ways described in comments by me and others to write a succinct annotation, and this will catch cases where somebody tries to use a dict keyed with strings or something.
Anyway, you don’t have to burn a lot of mental energy on them, they cost next to nothing at runtime, they help document your function signatures, and they help flag inconsistent assumptions in your codebase even if they’re not airtight. What’s not to like?
>from typing import Sequence
>def third(something: Sequence):
> return indexable[3]
however if all you are doing is just iterate over the thing, what you actually need is an Iterable
>from typing import Iterable
>def average(something:Iterable):
> for thing in something:
> ...
Statistically, the odds of a language being wrong, are much lower than the programmer being wrong. Not to say that there aren't valid critiques of python, but we must think of the creators of programming languages and their creations as the top of the field. If a 1400 chess elo player criticizes Magnus Carlsen's chess theory, it's more likely that the player is missing some theory rather than he found a hole in Carlsen's game, the player is better served by approaching a problem with the mentality that he is the problem, rather than the master.
The people at the top of the type-system-design field aren’t working on Python.
Sequence involves more than just __getitem__ with an int index, so if it really is anything int indexable, a lighter protocol with just that method will be more accurate, both ar conveying intent and at avoiding needing to evolve into an odd union type because you have something that a satisfies the function’s needs but not the originally-defined type.
No world class expert is going to contribute to Python after 2020 anyway, since the slanderous and libelous behavior of the Steering Council and the selective curation of allowed information on PSF infrastructure makes the professional and reputational risk too high. Apart from the fact that Python is not an interesting language for language experts.
Google and Microsoft have already shut down several failed projects.
I get the idea that Python and Java went in opposite directions. But I'm not aware of any fight between both languages. I don't think that's a thing either.
Regarding stuff that happens in the 2020. Python was developed in the 90s, python 3 was launched in 2008. Besides some notable PEPs like type hints, WSGI, the rest of development are footnotes. The same goes for most languages (with perhaps the exception of the evergrowing C++), languages make strong bc guarantees and so the bulk of their innovation comes from the early years.
Whatever occurs in the 20th and 30th year of development is unlikely to be revolutionary or very significant. Especially ignoreable is the drama that might emerge in these discussions, slander, libel inter-language criticism?
Just mute that out. I've read some news about some communities like Ruby on Rails or Nix that become overtaken by people and discussions of political nature rather than development, they can just be ignored I think.
Could you elaborate on this?
Before that, Google moved heavily from Python to Go.
Microsoft fired the "Faster CPython Team" this year.
For example the dart/flutter team was decimated as well.
That's your problem right there. Why are random callers sending whatever different input types to that function?
That said, there are a few existing ways to define that property as a type, why not a protocol type "Indexable"?
it was a sin that python's type system was initially released as a nominal type system. they should have been the target from day one.
being unable to just say "this takes anything that you can call .hello() and .world() on" was ridiculous, as that was part of the ethos of the dynamically typed python ecosystem. typechecking was generally frowned upon, with the idea that you should accept anything that fit the shape the receiving code required. it allowed you to trivially create resource wrappers and change behaviors by providing alternate objects to existing mechanisms. if you wanted to provide a fake file that read from memory instead of an actual file, it was simple and correct.
the lack of protocols made hell of these patterns for years.
Really it enabled the Python type system to work as well as it does, as opposed to TypeScript, where soundness is completely thrown out except for some things such as enums
Nominal typing enables you to write `def ft_to_m(x: Feet) -> Meters: and be relatively confident that you're going to get Feet as input and Meters as output (and if not, the caller who ignored your type annotations is okay with the broken pieces).
The use for protocols in Python in general I've found in practice to be limited (the biggest usefulness of them come from the iterable types), when dealing with code that's in a transitional period, or for better type annotations on callables (for example kwargs, etc).
Most Python's dunder methods make it so you can make "behave alike" objects for all kinds of behaviors, not just iterables
The current docs are "Microsoft-like", they have everything, spread through different pages, in different hierarchies, some of them wrong, and with nothing telling you what else exists.
Because it’s nice to reuse code. I’m not sure why anyone would think this is a design issue, especially in a language like Python where structural subtyping (duck typing) is the norm. If I wanted inheritance soup, I’d write Java.
Ironically, that’s support for structural subtyping is why Protocols exist. It’s too bad they aren’t better and the primary way to type Python code. It’s also too bad that TypedDict actively fought duck typing for years.
Python’s type system is overall pretty weak, but with any static language at least one of the issues is that the type system can’t express all useful and safe constructs. This leads to poor code reuse and lots of boilerplate.
This kind of accidental compatibility is a source of many hard bugs. Things appear to work perfectly, then at some point it does something subtly different, until it blows up a month later
Probably because the actual type it takes is well-understood (and maybe even documented in informal terms) by the people making and using it, but they just don’t understand how to express it in the Python type system.
You are looking for protocols. A bit futzy to write once but for a heavily trafficked function it's woth it.
If your JIT compiler doesn't work well with protocols... sounds like a JIT problem not a Python typing problem
sqlalchemy.orm.relationship(argument: _RelationshipArgumentType[Any] | None = None, secondary: _RelationshipSecondaryArgument | None = None, *, uselist: bool | None = None, collection_class: Type[Collection[Any]] | Callable[[], Collection[Any]] | None = None, primaryjoin: _RelationshipJoinConditionArgument | None = None, secondaryjoin: _RelationshipJoinConditionArgument | None = None, back_populates: str | None = None, order_by: _ORMOrderByArgument = False, backref: ORMBackrefArgument | None = None, overlaps: str | None = None, post_update: bool = False, cascade: str = 'save-update, merge', viewonly: bool = False, init: _NoArg | bool = _NoArg.NO_ARG, repr: _NoArg | bool = _NoArg.NO_ARG, default: _NoArg | _T = _NoArg.NO_ARG, default_factory: _NoArg | Callable[[], _T] = _NoArg.NO_ARG, compare: _NoArg | bool = _NoArg.NO_ARG, kw_only: _NoArg | bool = _NoArg.NO_ARG, lazy: _LazyLoadArgumentType = 'select', passive_deletes: Literal['all'] | bool = False, passive_updates: bool = True, active_history: bool = False, enable_typechecks: bool = True, foreign_keys: _ORMColCollectionArgument | None = None, remote_side: _ORMColCollectionArgument | None = None, join_depth: int | None = None, comparator_factory: Type[RelationshipProperty.Comparator[Any]] | None = None, single_parent: bool = False, innerjoin: bool = False, distinct_target_key: bool | None = None, load_on_pending: bool = False, query_class: Type[Query[Any]] | None = None, info: _InfoType | None = None, omit_join: Literal[None, False] = None, sync_backref: bool | None = None, **kw: Any) → Relationship[Any]Although Python type hints are not expressive enough.
interface IntIndexable {
[key: number]: any
}You can specify a protocol like this:
class IntIndexable(Protocol[T]):
def __getitem__(self, index: int, /) -> T: ...
Although I understant that it might have been just a simplified example. Usually the "Real World" can get very complex.
Yes it is. I believe the reason is that this is all valid python while typescript is not valid javascript. Also, python's type annotations are available at runtime (eg. for introspection) while typescript types aren't.
That said, typescript static type system is clearly both more ergonomic and more powerful than Python's.
No, this is the great thing about gradual typing! You can use it to catch errors and provide IDE assistance in the 90% of cases where things have well-defined types, and then turn it off in the remaining 10% where it gets in the way.
I would love it if it were better designed. It’s a real downer that you can’t check lots of Pythonic, concise code using it.
For broad things, write Any or skip it.
T_co = TypeVar("T_co", covariant=True)
class Indexable(Protocol[T_co]): def __getitem__(self, i: int) -> T_co: ...
def f(x: Indexable[str]) -> None: print(x[0])
I am failing to format it proprely here, but you get the idea.
> Text after a blank line that is indented by two or more spaces is reproduced verbatim. (This is intended for code.)
If you'd want monospace you should indent the snippet with two or more spaces:
from typing import Protocol, TypeVar
T_co = TypeVar("T_co", covariant=True)
class Indexable(Protocol[T_co]):
def __getitem__(self, i: int) -> T_co: ...
def f(x: Indexable[str]) -> None:
print(x[0])Generally it’s not worth trying to fix this stuff. The type signature is hell to write and ends up being super complex if you get it to work at all. Write a cast or Any, document why it’s probably ok in a comment, and move on with your life. Pick your battles.
Sequence[SupportsFloat] | Mapping[int,SupportsFloat]
This is really just the same mistake as the original expanding union, but with overly narrow abstract types instead of overly narrow concrete types. If it relies on “we can use indexing with an int and get out something whose type we don’t care about”, then its a Protocol with the following method:
def __getitem__(self, i: int, /) -> Any: ...
def __getitem__(self, i: int, /) -> T: ...
So, just:
class Indexable[T](Protocol):
def __getitem__(self, i: int,/) -> T: ...
If that is exactly what you want, then define a Protocol: from __future__ import annotations from typing import Protocol, TypeVar
T = TypeVar("T")
K = TypeVar("K")
class GetItem(Protocol[K, T]):
def __getitem__(self, key: K, /) -> T: ...
def first(xs: GetItem[int, T]) -> T:
return xs[0]
Can’t you just use a typing.Protocol on __getitem__ here?
https://typing.python.org/en/latest/spec/protocol.html
Something like
from typing import Protocol
class Indexable(Protocol):
def __getitem__(self, i: int) -> Self: ...
IMO, the trick to really enjoying python typing is to understand it on its own terms and really get comfortable with generics and protocols.
That being said, especially for library developers, the not-yet-existant intersection type [1] can prove particularly frustrating. For example, a very frequent pattern for me is writing a decorator that adds an attribute to a function or class, and then returns the original function or class. This is impossible to type hint correctly, and as a result, anywhere I need to access the attribute I end up writing a separate "intersectable" class and writing either a typeguard or calling cast to temporarily transform the decorated object to the intersectable type.
Also, the second you start to try and implement a library that uses runtime types, you've come to the part of the map where someone should have written HERE BE DRAGONS in big scary letters. So there's that too.
So it's not without its rough edges, and protocols and overloads can be a bit verbose, but by and large once you really learn it and get used to it, I personally find that even just the value of the annotations as documentation is useful enough to justify the added work adding them.
Change the declaration to:
def __getitem__(self, i: int | slice)
Though to be honest I am more concerned about that function that accepts a wild variety of objects that seem to be from different domains...
I'd guess inside the function is a HUGE ladder of 'if isinstance()' to handle the various types and special processing needed. Which is totally reeking of code smell.
That's not how you are supposed to use static typing? Python has "protocols" that allows for structural type checking which is intended for this exact problem.
The bigger problem is that the type system expressed through hints in Python is not the type system Python is actually using. It's not even an approximation. You can express in the hint type system things that are nonsense in Python and write Python that is nonsense in the type system implied by hints.
The type system introduced through typing package and the hints is a tribute to the stupid fashion. But, also, there is no syntax and no formal definitions to describe Python's actual type system. Nor do I think it's a very good system, not to the point that it would be useful to formalize and study.
In Russian, there's an expression "like a saddle on a cow", I'm not sure what the equivalent in English would be. This describes a situation where someone is desperately trying to add a desirable feature to an exiting product that ultimately is not compatible with such a feature. This, in my mind, is the best description of the relationship between Python's actual type system and the one from typing package.
“To fit a square peg into a round hole”
though maybe there's a path forward to give a variable a sort of "de-hint" in that in can be everything BUT this type(i.e. an argument can be any indexable type, except a string)
I think this is called a negation type, and it acts like a logical NOT operator. I'd like it too, and I hear that it works well with union types (logical OR) and intersection types (logical AND) for specifying types precisely in a readable way.
These are similar to interfaces in C# or traits in Rust - you describe what the parameter _does_ instead of what it _is_.
Mind you, I haven't used it before, but it feels very similar to the abstract Mapping types.
def lol(blarg): # types? haha you wish. rtfc you poor sod. Pytharn spirit ftw!!!
...
return omg[0].wtf["lol freedom"].pwned(Good.LUCK).figuring * outThis is a good way of expressing my own frustration with bolting strong typing on languages that were never designed to have it. I hate that TypeScript has won out over JavaScript because of this - it’s ugly, clumsy, and boilerplatey - and I’d be even more disappointed to see the same thing happen to the likes of Python and Ruby.
My background is in strongly typed languages - first C++, then Java, and C# - so I don’t hate them or anything, but nowadays I’ve come to prefer languages that are more sparing and expressive with their syntax.
Besides, there must be some behavior you expect from this object. You could make a type that reflects this: IntIndexable or something, with an int index method and whatever else you need.
This feels like an extremely weak argument. Just think of it as self-enforcing documentation that also benefits auto-complete; what's not to love? Having an IntIndexable type seems like a great idea in your use case.
And you better believe that every single admissible type
At my work we have a jit compiler that requires type hints under some conditions.
Aside from that, I avoid them as much as possible. The reason is that they are not really a part of the language, they violate the spirit of the language, and in high-usage parts of code they quickly become a complete mess.
For example a common failure mode in my work’s codebase is that some function will take something that is indexable by ints. The type could be anything, it could be List, Tuple, Dict[int, Any], torch.Size, torch.Tensor, nn.Sequential, np.ndarray, or a huge host of custom types! And you better believe that every single admissible type will eventually be fed to this function. Sometimes people will try to keep up, annotating it with a Union of the (growing) list of admissible types, but eventually the list will become silly and the function will earn a # pyre-ignore annotation. This defeats the whole point of the pointless exercise.
So, if the jit compiler needs the annotation I am happy to provide it, but otherwise I will proactively not provide any, and I will sometimes even delete existing annotations when they are devolving into silliness.