[or-cvs] r17299: {updater} Fix bundle download logic. Better fix needed. (in updater/trunk: . lib/thandy)

[nickm at seul.org]

Author: nickm
Date: 2008-11-16 18:47:50 -0500 (Sun, 16 Nov 2008)
New Revision: 17299

Modified:
   updater/trunk/TODO
   updater/trunk/lib/thandy/formats.py
   updater/trunk/lib/thandy/repository.py
Log:
Fix bundle download logic.  Better fix needed.

Modified: updater/trunk/TODO
===================================================================
--- updater/trunk/TODO	2008-11-16 23:35:08 UTC (rev 17298)
+++ updater/trunk/TODO	2008-11-16 23:47:50 UTC (rev 17299)
@@ -1,4 +1,11 @@
+- Bugs.
+  - Check hashes of downloaded packages and bundles as they are received.
 
+- Download improvements.
+  - Back off on download failure.
+  - Handle full stalled file in download.
+  - Use if-modified-since on timestamp
+
 - Write client-side code
   . Decide early if a python implementation will do for v1.
      o Adjust httplib, urllib2 to use socks4a.
@@ -20,7 +27,7 @@
 - Better configurability: let users override mirrors, keys, etc.
 - Backend for "A bunch of files you can unzip someplace."
 - OSX backend (hard).
-- Proper exponential backoff on backend.
+- Proper exponential back-off on download backend.
 
 - Wrapping
   - More unit tests

Modified: updater/trunk/lib/thandy/formats.py
===================================================================
--- updater/trunk/lib/thandy/formats.py	2008-11-16 23:35:08 UTC (rev 17298)
+++ updater/trunk/lib/thandy/formats.py	2008-11-16 23:47:50 UTC (rev 17299)
@@ -273,7 +273,7 @@
     """
     f_to_close = None
     if isinstance(f, basestring):
-        t_to_close = f = open(f, 'rb')
+        f_to_close = f = open(f, 'rb')
 
     useTempDigestObj = (digestObj == None)
     if useTempDigestObj:

Modified: updater/trunk/lib/thandy/repository.py
===================================================================
--- updater/trunk/lib/thandy/repository.py	2008-11-16 23:35:08 UTC (rev 17298)
+++ updater/trunk/lib/thandy/repository.py	2008-11-16 23:47:50 UTC (rev 17299)
@@ -390,7 +390,9 @@
                 continue
 
             rp = binfo.getRelativePath()
-            hashDict[rp] = h_expected = binfo.getHash()
+            #hashDict[rp] =  #XXXX this hash needs to be calculated ovver
+            #                #     the json data.
+            h_expected = binfo.getHash()
             bfile = self.getBundleFile(rp)
             try:
                 bfile.load()
@@ -401,7 +403,8 @@
 
             h_b = thandy.formats.getDigest(bfile.get())
             if h_b != h_expected:
-                logging.info("Bundle hash not as expected; must fetch.", rp)
+                logging.info("Bundle hash for %s not as expected; must fetch.",
+                             rp)
                 need.add(rp)
                 continue
 
@@ -422,7 +425,8 @@
                 rp = pkginfo['path']
                 pfile = self.getPackageFile(rp)
                 h_expected = thandy.formats.parseHash(pkginfo['hash'])
-                hashDict[rp] = h_expected
+                #hashDict[rp] =  #XXXX this hash needs to be calculated ovver
+                #                #     the json data.
                 try:
                     pfile.load()
                 except OSError:
@@ -477,7 +481,7 @@
                 fn = self.getFilename(rp)
                 try:
                     h_got = thandy.formats.getFileDigest(fn)
-                except OSError:
+                except (OSError, IOError):
                     logging.info("Installable file %s not found on disk; "
                                  "must load", rp)
                     need.add(rp)