django - Why is Heroku installing old Python (pip) dependencies in new deployments? -

django - Why is Heroku installing old Python (pip) dependencies in new deployments? -

(asking here directed heroku's own support)

we have uncovered dependency issue in project mismatched libraries between dev environments. details aren't relevant, underlying cause dependency had ">=" version match in setup.py - meant when dev rebuilt environment got latest version (0.4.0) instead of old version he'd had (0.3.11), , started getting deprecationwarning.

as part of debugging process, under impression whenever repo pushed heroku, clean environment rebuilt, led me assume, incorrectly, our dev environment (which rebuilt daily) have had latest version installed. because weren't seeing issue on dev environment decided investigate, , ran heroku run pip list on remote environment.

i (very) surprised see output of lucky dip of old , expired dependencies, , not clean environment @ all. turns out may have had issue debugging living happily on our live environment, part of old install.

the easiest way explain beautifulsoup library. updated v3 v4, , part of this, library changed name on pypi beautifulsoup beautifulsoup4. updated out requirements.txt reflect this, if run pip list on our heroku environment both:

~ $ heroku run bash ~ $ pip list beautifulsoup (3.2.1) beautifulsoup4 (4.3.2)

so, old dependency hasn't been cleared, it's sitting there. can prove plenty firing python session:

~ $ python python 2.7.4 (default, apr 6 2013, 22:14:13) [gcc 4.4.3] on linux2 type "help", "copyright", "credits" or "license" more information. >>> import bs4 >>> import beautifulsoup >>>

this bit of shock, , i'm amazed hasn't killed off our application @ point?

so, question - how heroku manage dependencies under hood - doesn't wipe python environment , re-run pip install on each deployment - , there way can forcefulness behaviour?

[edit 1]

fwiw, buildpack install - https://github.com/heroku/heroku-buildpack-python/blob/master/bin/compile

[edit 2]

from heroku buildpack docs:

the contents of cache_dir persisted between builds. can cache results of long processes dependency resolution here speed future builds.

and further down:

the bin/compile script given cache_dir sec argument can used store artifacts between builds. artifacts stored in directory available in cache_dir during successive builds. cache_dir available during slug compilation, , specific app beingness built.

the recommendation is:

heroku users can utilize heroku-repo plugin clear build cache created buildpack utilize app

that said - whilst understand cache used speed future compilations, don't understand why in cache installed. doesn't create sense?

heroku installs python environment in /app/.heroku/python , entire .heroku directory gets copied in cache_dir @ origin of each build , cache_dir @ end. (if search restore_cache , dump_cache in buildpack script you'll see lines this.)

so 1 time you've installed in heroku app, there stays unless cache_dir somehow gets wiped. isn't ideal, done avoid long build times result recompiling , installing dependencies on every deploy. (for it's worth, think there's improve way of achieving using wheels cache compiled packages can fresh environment every time without having wait rebuild. might seek set illustration , poke kenneth reitz it.)

in immediate term, sounds benefit explicitly pinning versions of dependencies (which include dependencies of dependencies , on). running pip freeze > requirements.txt start, although might want @ pip-tools. want avoid situation described dev rebuilds environment requirements.txt , ends different version of package.

as beautifulsoup changing bundle name (but not, presumably, name of module import) sounds horrible, , sort of situation in heroku's caching breaks down! think solution utilize heroku-repo plugin , nuke cache entirely.

python django heroku