Ticket #6805 (closed PLIP: fixed)
Integration of ZODB's blob support into Plone
Reported by: | witsch | Owned by: | witsch |
---|---|---|---|
Priority: | n/a | Milestone: | Ongoing |
Component: | General | Version: | |
Keywords: | blobs | Cc: | limi, rochael, forsberg, lucie, grahamperrin@… |
Description
ZODB 3.8 introduced support for blobs (large binary data) out of the box. Plone should utilize this new feature as support for large files without implications on RAM consumption etc has been a long-standing issue. As ZODB 3.8 is not included in Zope 2.10, this feature should probably target Plone 3.5, but compatibility with Plone 3.0 would be nice provided the newer ZODB version is used as well...
Attachments
Change History
comment:1 Changed 9 years ago by witsch
- Owner changed from somebody to witsch
- Status changed from new to assigned
comment:2 Changed 9 years ago by witsch
a short status update:
- the original implementation of ATBlobField was donated to the plone foundation by enfoldsystems and copied into the plone collective
- the product was updated quite a bit to play nicely as a drop-in replacement for archetypes' FileField
- the branch of ploneout for integration of blob support into plone was updated to incorporate ATBlobField
comment:3 Changed 9 years ago by witsch
after checking out the above mentioned branch of ploneout, the obligatory buildout and the necessary changes to zope.conf the following patch can be applied to make the standard "File" content type use ZODB blobs for all new instances:
-
products/ATContentTypes/content/file.py
33 33 34 34 from Products.Archetypes.atapi import Schema 35 from Products.Archetypes.atapi import FileField36 35 from Products.Archetypes.atapi import FileWidget 37 from Products.Archetypes.atapi import PrimaryFieldMarshaller38 36 from Products.Archetypes.atapi import AnnotationStorage 39 37 from Products.Archetypes.BaseContent import BaseContent … … 52 50 from Products.ATContentTypes import ATCTMessageFactory as _ 53 51 52 from Products.ATBlobField.field import BlobField, BlobMarshaller 53 54 54 from Products.validation.validators.SupplValidators import MaxSizeValidator 55 55 from Products.validation.config import validation … … 62 62 63 63 ATFileSchema = ATContentTypeSchema.copy() + Schema(( 64 FileField('file',64 BlobField('file', 65 65 required=True, 66 66 primary=True, … … 74 74 label=_(u'label_file', default=u'File'), 75 75 show_content_type = False,)), 76 ), marshall= PrimaryFieldMarshaller()76 ), marshall=BlobMarshaller() 77 77 ) 78 78 finalizeATCTSchema(ATFileSchema)
Changed 9 years ago by witsch
-
attachment
atct-file-using-blobs.patch
added
the above mentioned patch for convenience...
comment:4 follow-up: ↓ 5 Changed 9 years ago by drjnut
It doesn't work for small files. I didn't do many tests but >1MB works.
comment:5 in reply to: ↑ 4 ; follow-up: ↓ 6 Changed 9 years ago by witsch
Replying to drjnut:
It doesn't work for small files.
yes, for very small files the following error occurs:
Traceback (innermost last): Module ZPublisher.Publish, line 119, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 42, in call_object Module Products.CMFFormController.FSControllerPageTemplate, line 90, in __call__ Module Products.CMFFormController.BaseControllerPageTemplate, line 28, in _call Module Products.CMFFormController.ControllerBase, line 232, in getNext - __traceback_info__: ['id = base_edit', 'status = success', 'button=None', 'errors={}', 'context=<DemoFile at blob.2007-08-20.0582199210>', 'kwargs={}', 'next_action=None', ''] Module Products.CMFFormController.Actions.TraverseTo, line 38, in __call__ Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 42, in call_object Module Products.CMFFormController.FSControllerPythonScript, line 104, in __call__ Module Products.CMFFormController.Script, line 145, in __call__ Module Products.CMFCore.FSPythonScript, line 140, in __call__ Module Shared.DC.Scripts.Bindings, line 313, in __call__ Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec Module Products.CMFCore.FSPythonScript, line 196, in _exec Module None, line 1, in content_edit - <FSControllerPythonScript at /foo/content_edit used for /foo/blob.2007-08-20.0582199210> - Line 1 Module Products.CMFCore.FSPythonScript, line 140, in __call__ Module Shared.DC.Scripts.Bindings, line 313, in __call__ Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec Module Products.CMFCore.FSPythonScript, line 196, in _exec Module None, line 13, in content_edit_impl - <FSPythonScript at /foo/content_edit_impl used for /foo/blob.2007-08-20.0582199210> - Line 13 Module Products.Archetypes.BaseObject, line 655, in processForm Module Products.Archetypes.BaseObject, line 644, in _processForm - __traceback_info__: (<DemoFile at /foo/blob.2007-08-20.0582199210>, <Field file(file:rw)>, <bound method DemoFile.setFile of <DemoFile at /foo/blob.2007-08-20.0582199210>>) Module Products.Archetypes.utils, line 160, in mapply Module Products.Archetypes.ClassGen, line 76, in generatedMutator Module Products.ATBlobField.field, line 105, in set Module Products.ATBlobField.adapters.fileupload, line 24, in feed AttributeError: FileUpload instance has no attribute 'name'
I didn't do many tests but >1MB works.
the limit is actually 1000 bytes. as soon as the file size is 1001 or bigger, everything works as expected. the "problem" here is that python's cgi module handles small files differently, as can be seen in the __write function at line 691ff in cgi.py (in python 2.4.4):
def __write(self, line): if self.__file is not None: if self.__file.tell() + len(line) > 1000: self.file = self.make_file('') self.file.write(self.__file.getvalue()) self.__file = None self.file.write(line)
i'll try to add a workaround for this soon...
comment:7 follow-up: ↓ 8 Changed 8 years ago by witsch
a short sprint progress report: there are two flavours of blob buildouts now for your convenience, both of which using the new blob integration package:
- plone.app.blob/buildouts/ploneout is based on ploneout, so using mostly svn
- plone.app.blob/buildouts/plone-3.0 is based on plone.recipe.plone, so using the current plone3 tarball
comment:8 in reply to: ↑ 7 Changed 8 years ago by witsch
Replying to witsch:
a short sprint progress report: there are two flavours of blob buildouts now for your convenience, both of which using the new blob integration package:
for your convenience here are the commands to run these buildout flavours in order to get a running zope instance:
- for the flavour based on ploneout, i.e. targeted mainly to developers
svn co https://svn.plone.org/svn/plone/plone.app.blob/buildouts/ploneout python bootstrap.py ./bin/buildout ./bin/instance fg
- for the flavour based on plone.recipe.plone, i.e. targeted mainly to integrators
svn co http://svn.plone.org/svn/plone/plone.app.blob/buildouts/plone-3.0 python bootstrap.py ./bin/buildout ./bin/instance fg
comment:9 follow-up: ↓ 10 Changed 8 years ago by witsch
short status update: the above mentioned two flavours of blob buildouts don't use the special branch of zope 2.10 anymore, but work with the regular zope2 tarballs instead (with some monkey patching involved, of course :))
so, to try to use the integration in your existing plone 3.0 buildout (based on plone.recipe.plone) the following diff to your buildout.cfg as well as including the externals for PluggableAuthService and plone.recipe.zope2instance to fix pending bugs in these packages should be enough:
-
buildout.cfg
old new (this hunk was shorter than expected) 14 14 15 15 develop = 16 src/plone.recipe.zope2instance 16 17 17 18 [plone] … … 25 27 recipe = plone.recipe.zope2instance 26 28 zope2-location = ${zope2:location} 29 blob-storage = var/blobstorage 27 30 user = admin:admin 28 31 http-address = 8080 … … 33 36 ${buildout:eggs} 34 37 ${plone:eggs} 38 plone.app.blob 39 archetypes.schemaextender 40 Products.contentmigration 35 41 36 42 zcml = 43 plone.app.blob 44 archetypes.schemaextender-overrides 45 Products.contentmigration 37 46 38 47 products =
comment:10 in reply to: ↑ 9 Changed 8 years ago by witsch
Replying to witsch:
so, to try to use the integration in your existing plone 3.0 buildout (based on plone.recipe.plone) the following diff to your buildout.cfg as well as including the externals for PluggableAuthService and plone.recipe.zope2instance to fix pending bugs in these packages should be enough:
actually, it's still necessary to include plone.app.blob as a develop egg as well, since i haven't made a release yet... :)
comment:11 follow-ups: ↓ 12 ↓ 13 Changed 8 years ago by klm
i'm blocked trying to build both the ploneout and plone-3.0 buildouts, with this error:
[...] Getting distribution for 'zdaemon>=1.4a2,<1.4.999'. While: Installing instance. Getting distribution for 'zdaemon>=1.4a2,<1.4.999'. Error: Couldn't find a distribution for 'zdaemon>=1.4a2,<1.4.999'.
i've tried fiddling with the recipe's zdaemon version (and that of ZConfig, which was getting a similar error), but cannot arrive at one that both builds successfully and runs:
- 'zdaemon>=1.4a2,<1.9.999' cannot be satisfied
- 'zdaemon>=1.4a2,<2.0' yields zdaemon-20a6, but then
trying to start the system results in an error in zdctrl.py:
"AttributeError: 'dict' object has no attribute 'mapping'".)
(that's with a ZConfig entry 'ZConfig >=2.4a2,<3.0', which yields ZConfig 2.5)
so i'm stuck. (i'm working with an earlier version of plone.app.blob that i got working as a zeocluster, and wanted to release instructions for setting up the cluster with the current configuration, but can't until i can get past this.)
comment:12 in reply to: ↑ 11 ; follow-up: ↓ 15 Changed 8 years ago by klm
Replying to myself - klm:
i'm blocked trying to build both the ploneout and plone-3.0 buildouts, with this error:
i've succeeded in using the plone.recipe.plone method, instead, to get a working instance with blobs. being unacquainted with buildouts, there was a lot i had to infer to do so. to help those in a similar predicament (because the more exercising blobs, the better), and to get corrections where i missed an easier way, here's what i did:
- use the plone.org buildout tutorial instructions to install ZopeSkel / paster, if you don't already have it. (see the prerequesites and creating a buildout... sections).
- use paster to create a plone3_buildout: paster create -t plone3_buildout Plone-3_0_3-Blobs
- adjust the buildout, per andi's instructions in this issue (but there's a lot of inferrence to do here):
- apply the diff suggested in comment 9
- add src/plone.app.blob to the develop egg section, as sorta suggested in comment 10
- do svn checkouts of PluggableAuthService, plone.recipe.zope2instance, and plone.app.blob, as implied in the commentary of comment 9. i got the svn checkout addresses from the linked-to EXTERNALS.txt files - i wouldn't be surprised if i could have somehow used those files instead of doing manual checkouts - there's a lot layers involved here.
cd products svn co svn://svn.zope.org/repos/main/Products.PluggableAuthService/branches/1.5/Products/PluggableAuthService PluggableAuthService cd ../src svn co https://svn.plone.org/svn/plone/plone.app.blob/trunk plone.app.blob svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/trunk plone.recipe.zope2instance
- do the bootstrap and buildout:
python2.4 bootstrap.py bin/buildout -v
then you'll be able to run as a standalone instance, and when you create a plone site, select the plone.app.blob extension profile so blobs are an available content type. (it's not hard to substitute them for regular files, using the portal_types tool. Images and audio/video are yet to be done.) it looks like getting a zeo cluster going will take similar measures as it did for the ploneout or plone-3.0 integration packages - i plan to test that and make the info available, soon.
comment:13 in reply to: ↑ 11 ; follow-up: ↓ 14 Changed 8 years ago by witsch
Replying to klm:
i'm blocked trying to build both the ploneout and plone-3.0 buildouts, with this error:
hi ken,
that's probably my fault. i was just gonna re-try things with a fresh checkout and about to move my ~/.buildout/default.cfg out of the way when i noticed i have the following line in there:
index = http://download.zope.org/ppix
that's very likely the reason buildout found the eggs for me, but not generically, of course. i'm just double checking atm...
comment:14 in reply to: ↑ 13 Changed 8 years ago by witsch
Replying to witsch:
that's very likely the reason buildout found the eggs for me, but not generically, of course. i'm just double checking atm...
yep, r18085 should fix this problem, so "svn up" please. and sorry about that — rather stupid oversight on my part. actually, i was wondering a bit why merging hanno's r18009 (in r18045) wouldn't cause any of the egg to be not found as i was sort of expecting... :)
comment:15 in reply to: ↑ 12 ; follow-ups: ↓ 16 ↓ 84 Changed 8 years ago by witsch
Replying to klm:
i've succeeded in using the plone.recipe.plone method, instead, to get a working instance with blobs. being unacquainted with buildouts, there was a lot i had to infer to do so.
sorry for the hassle again, ken. like i said, that was a pretty silly mistake to do... :(
to help those in a similar predicament (because the more exercising blobs, the better),
+1 and thanks for the workaround — i hope it shouldn't be necessary to go through all these manual steps anymore, though *duck*. i mean, that was the basic idea of providing the buildouts in the first place, but of course this is still badly lacking documentation...
however, i hope to be able to do a first release to the cheeseshop soon (i.e. tomorrow), which will tell about what to do to get blobs working in a more structured manner.
- use paster to create a plone3_buildout:
yep, the buildout.cfg you're getting from that will contain the additional helper links from r18085.
(it's not hard to substitute them for regular files, using the portal_types tool. Images and audio/video are yet to be done.)
you can also try to point your browser to .../@@migrate-files-to-blobs to migrate all you existing File content to Blob instances, but i'd highly recommend backing up your Data.fs beforehand at this point... :)
comment:16 in reply to: ↑ 15 ; follow-up: ↓ 17 Changed 8 years ago by klm
Replying to witsch:
Replying to klm:
i've succeeded in using the plone.recipe.plone method, instead, to get a working instance with blobs. being unacquainted with buildouts, there was a lot i had to infer to do so.
sorry for the hassle again, ken. like i said, that was a pretty silly mistake to do... :(
to help those in a similar predicament (because the more exercising blobs, the better),
+1 and thanks for the workaround i hope it shouldn't be necessary to go through all these manual steps anymore, though *duck*. i mean, that was the basic idea of providing the buildouts in the first place, but of course this is still badly lacking documentation...
thanks, andi! i'm more confused, though!-) i tried a few things to obtain your new revisions, but apparently don't understand how i'm supposed to get them. i expect that my answer for now is to work with my above-obtained build, and look forward to replacing that with...
however, i hope to be able to do a first release to the cheeseshop soon (i.e. tomorrow), which will tell about what to do to get blobs working in a more structured manner.
cool! thanks...
comment:17 in reply to: ↑ 16 ; follow-up: ↓ 18 Changed 8 years ago by witsch
Replying to klm:
thanks, andi! i'm more confused, though!-)
oh, that's not what i was hoping to read... :)
i tried a few things to obtain your new revisions, but apparently don't understand how i'm supposed to get them.
actually, you should now be able to use one of the sets of 4 commands listed in my previous comment to get a working instance with blob support; however, i've noticed the "cd ploneout" or "cd plone-3.0" are missing (as the second line in each set)
but i reckon you should also be able to keep using your own buildout and just do an
$ svn up
in the main directory of that; and we should also be able to figure things out "interactively" on #plone if nothing else...
comment:18 in reply to: ↑ 17 ; follow-up: ↓ 19 Changed 8 years ago by klm
Replying to witsch:
Replying to klm:
thanks, andi! i'm more confused, though!-)
oh, that's not what i was hoping to read... :)
i tried a few things to obtain your new revisions, but apparently don't understand how i'm supposed to get them.
actually, you should now be able to use one of the sets of 4 commands listed in my previous comment to get a working instance with blob support; however, i've noticed the "cd ploneout" or "cd plone-3.0" are missing (as the second line in each set)
ah, that was the source of my confusion. i was thinking that the paster of plone3_buildout was the preferred method, and was reluctant to head back to the svn checkouts - but should have realized that's the expected thing to do (and that the paster buildout uses released, not frontier, versions). using an svn checkout will be good - thanks for the clarification.
comment:19 in reply to: ↑ 18 Changed 8 years ago by witsch
Replying to klm:
ah, that was the source of my confusion. i was thinking that the paster of plone3_buildout was the preferred method,
well yes, it is. but you'd only really want to use paster to create your own buildouts — in this case i've already done that for you, so you just need to check it out and run it. that was the basic idea anyway... :) but as a side note, the plone-3.0 buildout was created using paster, of course (see r17624).
and was reluctant to head back to the svn checkouts -
right, but in this case the modified (to the needs for blob support) buildout is provided for your convenience in subversion again. that saves you from creating a fresh buildout (with paster) and applying all the needed modifications (as you're now familiar with :)) as i've already set that up for you.
so instead of having to do several svn checkouts like in the past, you only need one now — essentially you're just getting buildout.cfg and bootstrap.py with the plone.recipe.plone-based flavour. that'd be the ideal scenario anyway — atm the buildout does need some svn:externals, because of some bugfixes that haven't been released in new versions yet. and the egg of plone.app.blob itself is also still missing, of course...
what might have been a little confusing, though, was my comment about how to integrate blob support into your own buildouts. that was meant for people with already existing and customized buildouts, i.e. those who'd like to give blob support a spin on an existing project.
but should have realized that's the expected thing to do (and that the paster buildout uses released, not frontier, versions). using an svn checkout will be good - thanks for the clarification.
yep, atm the checkout's easier, because it automatically gets those still needed and yet unreleased versions for you via svn:externals. once they're released things would ideally boil down to adding the plone.app.blob egg to your buildout.cfg, which would make using paster just as simple again...
comment:20 Changed 8 years ago by witsch
status update: since r18201 it's now actually possible to run the CMFPlone tests using the new blob content type as a drop-in replacement for ATFile with a blob-enabled zodb storage (in the testrunner). to do this you need to patch plone's test base class as follows:
-
products/CMFPlone/tests/PloneTestCase.py
5 5 # $Id$ 6 6 7 from plone.app.blob import db # needs to be imported first to set up ZODB 7 8 from Products.PloneTestCase.ptc import * 8 9 … … 13 15 classImplements(PloneSite, ITestCasePloneSiteRoot) 14 16 15 setupPloneSite(extension_profiles=['Products.CMFPlone:testfixture']) 17 from plone.app.blob.tests import setupPackage 18 setupPackage() 19 20 setupPloneSite(extension_profiles=['Products.CMFPlone:testfixture', 'plone.app.blob:testing']) 16 21
i'll try to make this somewhat more convenient, but only after the tests pass again... :)
Changed 8 years ago by witsch
-
attachment
cmfplone-tests-with-blob-as-atfile-dropin.patch
added
the patch mentioned in the last comment for your convenience…
comment:21 Changed 8 years ago by witsch
another update: all CMFPlone tests are passing after r18220... time to go to bed! :)
comment:22 follow-up: ↓ 23 Changed 8 years ago by klm
looks like a file checkin may be missing, andi - trying to start with a buildout from yesterday evening and this afternoon yeilds (a long traceback ending in):
IOError: [Errno 2] No such file or directory: '/home/klm/Servers/Builds/Ploneout+Blobs/addons/archetypes.schemaextender/archetypes/schemaextender/overrides.zcml'
creating a trivial overrides.zcml in the indicated location (with just the <configure> element) enables me to start - but likely isn't the right content.
comment:23 in reply to: ↑ 22 Changed 8 years ago by witsch
Replying to klm:
looks like a file checkin may be missing, andi -
that's because martin's already made the change to archetypes.schemaextender we had discussed, but i had not noticed and changed my buildout accordingly yet... r18233 should do this, so an "svn up" should get you going again.
creating a trivial overrides.zcml in the indicated location (with just the <configure> element) enables me to start - but likely isn't the right content.
actually it is, since it's gone now, which is sort of like empty... :) but now the previously empty and therefore not needed configure.zcml has some content and needs to be included.
comment:24 Changed 8 years ago by witsch
- Cc optilude added
update: after r18236 the buildout based on plone.recipe.plone is not quite svn:externals-free yet (i.e. "egg-only"), but slowly getting there... perhaps optilude will have some spare time to make a new alpha release of archetypes.schemaextender soon enough? :)
comment:25 follow-up: ↓ 26 Changed 8 years ago by klm
as threatened, i've created a web page describing how to use andi's work to build a blob-aware plone site. further, it details how to adjust the buildout and do some manual editing of the result to create a working, blob-aware plone cluster (gotta have zeo:-). i think this will help identify recipe changes to get that working without the edits.
i plan to run through the instructions again to shake out typos and omissions. problem reports and suggestions are welcome...
comment:26 in reply to: ↑ 25 ; follow-up: ↓ 28 Changed 8 years ago by witsch
Replying to klm:
as threatened, i've created a web page describing how to use andi's work to build a blob-aware plone site.
heh, i've read an early version i think friday night or so, then reloaded the page twice during the weekend, one time giving me "last edited 10 seconds ago", the second "last edited 6 minutes ago", so it sure looked like you were still working on it.
unfortunately, i haven't gotten around to catch up with the updates today, but then again atm it's "last edited 14 minutes ago", so i'm not sure if i should wait a little longer — could you perhaps let me know once you think it's converging to 1.0? :)
i think this will help identify recipe changes to get that working without the edits.
yep, it will, and that's so helpful. thanks for putting this together, ken! oh, and btw, i didn't mention above that the early draft i've read already looked pretty good as a blob howto, and a very detailed one so... :)
comment:27 Changed 8 years ago by witsch
- Cc optilude removed
oh, and i guess martin's got the message, so i'll reduce his spam again (somewhat)... :)
comment:28 in reply to: ↑ 26 ; follow-ups: ↓ 29 ↓ 30 Changed 8 years ago by klm
Replying to witsch:
Replying to klm:
as threatened, i've created a web page describing how to use andi's work to build a blob-aware plone site. [...]
unfortunately, i haven't gotten around to catch up with the updates today, but then again atm it's "last edited 14 minutes ago", so i'm not sure if i should wait a little longer — could you perhaps let me know once you think it's converging to 1.0? :)
heh - i should have said, that was my plan.
i just got running with effective-user working, and that's the last of the problems in the things i can think to exercise. i may have scrambled the instructions a bit, in the process of shaking out the later problems - we can handle that as it emerges.
i think this will help identify recipe changes to get that working without the edits.
yep, it will, and that's so helpful. thanks for putting this together, ken! oh, and btw, i didn't mention above that the early draft i've read already looked pretty good as a blob howto, and a very detailed one so... :)
i'm easily confused by ambiguity, so i'm a little extreme about trying to be clear when i write. hope i didn't overdo it.
anyway, i'd say it's cooked. ask when you hit unclear spots...
comment:29 in reply to: ↑ 28 ; follow-up: ↓ 31 Changed 8 years ago by klm
Replying to klm:
Replying to witsch:
Replying to klm:
as threatened, i've created a web page describing how to use andi's work to build a blob-aware plone site. [...]
unfortunately, i haven't gotten around to catch up with the updates today, but then again atm it's "last edited 14 minutes ago", so i'm not sure if i should wait a little longer — could you perhaps let me know once you think it's converging to 1.0? :)
heh - i should have said, that was my plan.
oh, yeah - i should also have said, i added little "status" notes to the contents list in the intro, at top. i plan to add notes indicating what changed if i discover more things in the already established stuff. conversely, i'll add notes for the later sections (arranging for new files and images to be blob-based, and implementing blob-based video and audio, once i start filling them in. that might be within a few days.
comment:30 in reply to: ↑ 28 Changed 8 years ago by witsch
Replying to klm:
Replying to witsch:
oh, and btw, i didn't mention above that the early draft i've read already looked pretty good as a blob howto, and a very detailed one so... :)
i'm easily confused by ambiguity, so i'm a little extreme about trying to be clear when i write. hope i didn't overdo it.
no, don't get me wrong — i think that's a good thing!
anyway, i'd say it's cooked. ask when you hit unclear spots...
alright, it looks like i won't have time today as i can only work for about 2.5 hours and i want to get the fixes done, but i'll have a look tomorrow...
comment:31 in reply to: ↑ 29 ; follow-up: ↓ 32 Changed 8 years ago by witsch
Replying to klm:
conversely, i'll add notes for the later sections (arranging for new files and images to be blob-based, and implementing blob-based video and audio, once i start filling them in.
the former should be really easy, at least for files. there's a gs profile modifying the "File" FTI so that the new blob content type is used instead. it's currently registered in a way so that it's only available for testing, cause this is what i'm using it for, but it'd be really easy to change that so that one could just apply the profile via /portal_setup and get blobs instead of atfiles. what do you think?
comment:32 in reply to: ↑ 31 ; follow-up: ↓ 37 Changed 8 years ago by klm
Replying to witsch:
Replying to klm:
conversely, i'll add notes for the later sections (arranging for new files and images to be blob-based, and implementing blob-based video and audio, once i start filling them in.
the former should be really easy, at least for files. there's a gs profile modifying the "File" FTI so that the new blob content type is used instead. it's currently registered in a way so that it's only available for testing, cause this is what i'm using it for, but it'd be really easy to change that so that one could just apply the profile via /portal_setup and get blobs instead of atfiles. what do you think?
sounds good! plus, it clues me in to GenericSetup (though it took a bit to figure out that's what "gs" is). are you thinking about implementing this, soon? if not, i'll try to fill in the blanks, though it'd be great if you'd do so, if it would be easy.
once that's done, i expect that i can easily implement something for images, basing it on what's done for files with subtypes/file.py in the plone.app.blob addon.
after that, i imagine it's only a slightly larger step to do something similar for a new audio/video media content type, crafting a schema that provides for an additional metadata field (a place-holder picture). i already have some working skin templates, adapted from an older version of the plone4artists audio/video type, ATVideo, which provide in-browser players adapted to the underlying audio/video medium type. i've used those templates in my previous iteration, inhibiting most or all the other ATVideo machinery because it's there basically to provide a workaround version of blobs...
that's where i'm thinking of heading with this, and there doesn't seem to have any hard pieces. i do have one hesitation, in a design choice, but that i can investigate that while continuing with this course.
that question concerns why a content-type distinction is made at all between underlying binary media. it's tempting to have just one blob-based file type (perhaps with an additional picture place-holder metadata, as i'd do for audio/video) and have the browser templates discriminate how to present the file - as an image, audio player, video player, animation player (flash, etc), pdf, or non-player file. this seems like the simplest thing, UI wise, putting all the presentation smarts in the presentation machinery. it also seems quite justifiable, conceptually - if you're adding some not-directly-editable media, add it as a media file. moreover, as more players arrive on the scene, the players can be back-fitted to existing files of the corresonding media type.
the only potential drawback i see is for categorizing, eg picture collections versus video collections, etc - but that kind of distinction can be based on file metadata, rather than content type, couldn't it?
that question is the one i've been burning to ask for a while, now, but i haven't known who to ask - have you and or alexander questioned the traditional design of distinct content types (having a file versus image versus video versus audio), and/or is there an obvious reason that i'm missing to go this way?
if there's not an easy answer to this question, i can just go with the current design, trying to find the right place to address the question in parallel.
comment:33 Changed 8 years ago by witsch
quick status update before tackling webdav (and answering ken's comment, but probably not before tomorrow): r18321 adds the patches necessary to run the ATCT tests, so you guys can review and also run them yourselves. thanks to a couple of other fixes the only remaining failures are now in webdav.txt ...
comment:34 Changed 8 years ago by witsch
r18322:18323 fix the remaining issues, so that now all CMFPlone and ATCT tests pass, yay! a good time to go to bed... :)
comment:35 follow-up: ↓ 38 Changed 8 years ago by klm
the buildout command fails with the updated stuff because of a missing cache-size entry in the [common] section of my base.cfg. i've added the line to my online version of base.cfg, which you can either re-copy or just make the change by hand and check it in. the buildout then works. (it's great to see the changes in place, and get to exercise them - thanks!) i ran out of time (once comcast got my neighborhoods internet connection fixed) to apply and exercise my instructions all the way through, but will do that next chance i get.
comment:36 follow-up: ↓ 39 Changed 8 years ago by andycat
Will this be available for plone 2.5.x ?
comment:37 in reply to: ↑ 32 Changed 8 years ago by witsch
Replying to klm:
sounds good! plus, it clues me in to GenericSetup (though it took a bit to figure out that's what "gs" is).
sorry :)
are you thinking about implementing this, soon? if not, i'll try to fill in the blanks, though it'd be great if you'd do so, if it would be easy.
should be pretty trivial. in fact, the profile's already there for testing purposes — just not exposed to the user yet...
once that's done, i expect that i can easily implement something for images, basing it on what's done for files with subtypes/file.py in the plone.app.blob addon.
well, just replacing the field in ATImage shouldn't be that hard i suppose, but actually re-creating its behaviour as a subtype will bring up some obstacles, imho. however, i guess it's worth the effort nevertheless, since it'll probably clean up things a lot — just like it did for the ATFile replacement. imho that code is much leaner and also cleaner now and still all tests are passing...
after that, i imagine it's only a slightly larger step to do something similar for a new audio/video media content type, crafting a schema that provides for an additional metadata field (a place-holder picture).
yup, that should actually be quite easy, cause afaik the p4a products rely on ATFile and ATImage as their base. so once the basic sub-types are in shape, integration the p4a layer shouldn't be overly complicated, hopefully... :)
that question concerns why a content-type distinction is made at all between underlying binary media. it's tempting to have just one blob-based file type
right, and that's exactly where we're aiming at for plone 4.0, at least that's what everyone agreed upon in naples... ;) it doesn't make sense to have several content types for binary data, of course.
and have the browser templates discriminate how to present the file - as an image, audio player, video player, animation player (flash, etc), pdf, or non-player file.
basically that's where the sub-typing approach helps: you don't really want to put all of that into the same set of templates — that'd be way too messy and not extensible either. instead what sub-typing does is using dynamic marker interfaces and views/temlates/schema-extensions etc registered for those marker interfaces. then you decide, manually or automatically at upload or in whatever way, which "behaviour" you want and mark the blob item accordingly...
moreover, as more players arrive on the scene, the players can be back-fitted to existing files of the corresonding media type.
right, you'd just add another view in that case.
the only potential drawback i see is for categorizing, eg picture collections versus video collections, etc - but that kind of distinction can be based on file metadata, rather than content type, couldn't it?
sure, and it probably should anyway. otoh there will be the need to categorize on your assigned "content type", i.e. the sub-type or behaviour as we call it, but that's also easily done by indexing the marker interfaces each object holds.
have you and or alexander questioned the traditional design of distinct content types (having a file versus image versus video versus audio),
yes, i think we thought about that... :)
and/or is there an obvious reason that i'm missing to go this way?
to not to go this way you mean, no? but no, there isn't, imho.
comment:38 in reply to: ↑ 35 Changed 8 years ago by witsch
Replying to klm:
the buildout command fails with the updated stuff because of a missing cache-size entry in the [common] section of my base.cfg.
hmm, [common] is using the plone.recipe.zope2instance recipe, which should have a default for the cache-size, no? or rather for either zodb-cache-size or zeo-client-cache-size.
i've added the line to my online version of base.cfg,
i didn't find anything with the string "cache" in that page. have you fixed the problem in the meantime or am i missing something?
comment:39 in reply to: ↑ 36 ; follow-up: ↓ 42 Changed 8 years ago by witsch
Replying to andycat:
Will this be available for plone 2.5.x ?
yes, that's planned as the next step after the alpha release, i.e. i'll probably be able to start working on that next week...
comment:40 Changed 8 years ago by witsch
just fyi, the first alpha has been officially released to the PSC and the cheeseshop — announcement to follow...
comment:41 follow-up: ↓ 43 Changed 8 years ago by witsch
plus, there are new releases of archetypes.schemaextender and plone (yay!), so the buildout finally doesn't need any svn:externals anymore... :)
comment:42 in reply to: ↑ 39 ; follow-up: ↓ 47 Changed 8 years ago by klm
Replying to witsch:
Replying to andycat:
Will this be available for plone 2.5.x ?
yes, that's planned as the next step after the alpha release, i.e. i'll probably be able to start working on that next week...
yay on the alpha release!
i think i understand this priority, but want to be sure i understand - does this mean making the ATFile-based-on-blobs profile (adapted from the one you're using for testing) the default, and implementing images/etc as blob-based subtypes, all will come after the 2.5 implementation? and maybe a good bit later?
i'm trying to get a sense of where to aim my sights - if there's a lot of work to get the subtypes implementation done, maybe i'd better just go with the simpler (i presume:-) blob-instead-of-file for each of the binary content types (file, image, audio/video), and wait for the full subtyping implementation like everyone else. does that sound about right? (the mentions i've seen here and there, of plone 4.0 for the full, subtypes-based implementation suggests as much...)
comment:43 in reply to: ↑ 41 ; follow-ups: ↓ 44 ↓ 45 Changed 8 years ago by klm
- Type changed from enhancement to Bug
Replying to witsch:
plus, there are new releases of archetypes.schemaextender and plone (yay!), so the buildout finally doesn't need any svn:externals anymore... :)
my ploneout-based ( https://svn.plone.org/svn/plone/plone.app.blob/buildouts/ploneout), buildout now fails. this is from a checkout done a few minutes ago, and without any custom tailoring. the error is:
[...] We have the best distribution that satisfies 'setuptools'. Picked: setuptools = 0.6c7 While: Installing instance. Error: There is a version conflict. We already have: archetypes.schemaextender 1.0dev but plone.app.blob 1.0a1dev-r18422 requires 'archetypes.schemaextender>=1.0b1'.
i'm able to get the buildout to complete by editing addons/plone.app.blob/setup.py so that the install_requires entry for schemaextender looks like:
'archetypes.schemaextender >=1.0dev',
the resulting, completed build seems to work, blobs and all. still, i am not confident that this is the right fix. any idea what's going on?
comment:44 in reply to: ↑ 43 Changed 8 years ago by klm
- Type changed from Bug to Enhancement
Replying to klm:
whoops - i shouldn't have changed the type property to bug - i thought that was for the individual comment, not the issue as a whole. so i'm changing it back with this - but the issue is still as described in comment 43.
comment:45 in reply to: ↑ 43 ; follow-up: ↓ 46 Changed 8 years ago by witsch
Replying to klm:
my ploneout-based ( https://svn.plone.org/svn/plone/plone.app.blob/buildouts/ploneout), buildout now fails.
r18427 should fix this for now — i'll investigate later...
comment:46 in reply to: ↑ 45 Changed 8 years ago by witsch
comment:47 in reply to: ↑ 42 Changed 8 years ago by witsch
Replying to klm:
i think i understand this priority, but want to be sure i understand - does this mean making the ATFile-based-on-blobs profile (adapted from the one you're using for testing) the default, and implementing images/etc as blob-based subtypes, all will come after the 2.5 implementation?
as for the profile, no — r18438 has that change. however, it doesn't make it default, but optional. i'll update the docs asap, but will probably try to look into a few bugs (see TODO.txt) so i can make a second alpha including that change.
apart from that, yes, image support will only come after the backport to plone 2.5, so that might still take a little while depending on how well that goes... sorry.
if there's a lot of work to get the subtypes implementation done, maybe i'd better just go with the simpler (i presume:-) blob-instead-of-file for each of the binary content types (file, image, audio/video),
"blob-instead-of-file" is unlikely to work well for images atm. but you're most welcome to try and report what's missing, of course. this would help a lot, in fact.
the other thing you could do in the meantime (i mean, before images are supported through sub-typing), is replace the primary field of ATImage with the blob-based field, similar to one of the initial comments from above...
(the mentions i've seen here and there, of plone 4.0 for the full, subtypes-based implementation suggests as much...)
well, image support will be ready way before 4.0 — i hope to get this done before or at the snow sprint, which would make it even before 3.1, but we'll have to see...
comment:48 Changed 8 years ago by witsch
status update: the plip reviews have kept me busy last week, but with r18492 a buildout for plone 2.5 now finally starts up at least...
$ svn co https://svn.plone.org/svn/plone/plone.app.blob/buildouts/plone-2.5 blob-2.5 [...snip...] Checked out revision 18492. $ cd blob-2.5 $ python bootstrap.py Creating directory '/opt/zope/blob-2.5/bin'. Creating directory '/opt/zope/blob-2.5/parts'. Creating directory '/opt/zope/blob-2.5/develop-eggs'. Generated script '/opt/zope/blob-2.5/bin/buildout'. $ ./bin/buildout Develop: '/opt/zope/blob-2.5/src/plone.app.blob' Installing plone. Installing zope2. Installing productdistros. Installing fakezope2eggs. Installing instance. Generated script '/opt/zope/blob-2.5/bin/instance'. Generated script '/opt/zope/blob-2.5/bin/repozo'. Installing zopepy. Generated interpreter '/opt/zope/blob-2.5/bin/zopepy'. $ ./bin/instance fg "/opt/zope/blob-2.5/parts/instance/bin/runzope" 2007-12-18 15:11:26 INFO ZServer HTTP server started at Tue Dec 18 15:11:26 2007 Hostname: 0.0.0.0 Port: 8080 [...snip...] 2007-12-18 15:11:49 INFO Zope Ready to handle requests
that said, adding a plone site with the "plone.app.blob" profile doesn't work yet, of course... ;)
comment:49 follow-up: ↓ 50 Changed 8 years ago by rochael
- Cc rochael added
I'm working on an enhancement to plone.recipe.zope2zeoserver so it can generate blob storage configuration AND work off of ZDaemon and ZODB eggs instead of requiring a zope2 installation.
If any one wants to coordinate, please leave a reply.
I intend to commit my works to a branch for evaluation and then merge it back to the recipe trunk.
comment:50 in reply to: ↑ 49 ; follow-ups: ↓ 52 ↓ 53 Changed 8 years ago by klm
Replying to rochael:
I'm working on an enhancement to plone.recipe.zope2zeoserver so it can generate blob storage configuration AND work off of ZDaemon and ZODB eggs instead of requiring a zope2 installation.
that's great to hear! i've been wondering where and how to ask for the zope2zeoserver (and zope2instance) corrections for the problems that my PloneBlobs instructions workaround. i was thinking about submitting a trac ticket besides this one, but it sounds like you'll be addressing at least some of them. it would probably make sense for you to submit a trac ticket against plone.recipe.zope2zeoserver - that way, whoever feels some responsibility for zope2zeoserver (if anyone) would be alerted, and we would have a place to coordinate.
If any one wants to coordinate, please leave a reply.
all i can offer are the faults and corrections that i noticed - let me know if you need more details or consolidation of what's in the PloneBlobs instructions.
I intend to commit my works to a branch for evaluation and then merge it back to the recipe trunk.
i can probably test that, when you're ready. let us know here what the trac ticket is, if you create one, and include me among the Cc's, if you do.
comment:51 Changed 8 years ago by hannosch
The bug tracker for all the buildout recipes lives at https://bugs.launchpad.net/collective.buildout/. Feel free to add tickets and patches to it :)
comment:52 in reply to: ↑ 50 Changed 8 years ago by witsch
Replying to klm:
Replying to rochael:
I'm working on an enhancement to plone.recipe.zope2zeoserver so it can generate blob storage configuration AND work off of ZDaemon and ZODB eggs instead of requiring a zope2 installation.
cool! :)
i can probably test that, when you're ready. let us know here what the trac ticket is, if you create one, and include me among the Cc's, if you do.
me too, please.
comment:53 in reply to: ↑ 50 Changed 8 years ago by klm
Replying to klm:
Replying to rochael:
I'm working on an enhancement to plone.recipe.zope2zeoserver so it can generate blob storage configuration AND work off of ZDaemon and ZODB eggs instead of requiring a zope2 installation.
[...] all i can offer are the faults and corrections that i noticed - let me know if you need more details or consolidation of what's in the PloneBlobs instructions.
i've added a few launchpad collective.buildout bug reports:
- plone.recipe.zope2zeoserver generates conf disregarding blob-storage
- plone.recipe.zope2zeoserver ?generates? zeoctl and runzeo with incorrect PYTHONPATH and ZODB3_HOME
- plone.recipe.zope2instance zeo zope.conf neglects blob-storage setting
(while you didn't say you were going to work on plone.receipe.zope2instance, it also needs repair.)
some other things to keep in mind:
- if i understand christian theune's Blobs Zero Copy work correctly, much better performance is possible when the blobstorage directory for the server and clients should be the same when they reside on the same machine, so it would be nice if the recipe defaults arranged for that.
- the var directories need to be created, and the person doing the install needs to be advised about setting the group ownership to that of effective user, if the effective user is set. (it may even be that the user ownership needs to be set - but that requirement may be another bug...)
thanks, hannosch, for the lead on the launchpad topic.
comment:54 follow-up: ↓ 56 Changed 8 years ago by rochael
Arranging for the blob directories to be the same is just a matter of reusing the configuration variable from one section to the other:
[zeoserver] recipe = plone.recipe.zope2zeoserver zope2-location = ${zope2:location} blob-storage = ${buildout:directory}/var/blobstorage zeo-address = 8100 eggs = ZODB3 ... [instance] ... blob-storage = ${zeoserver:blob-storage} ...
The recipes need not concern themselves with that, though we should obviously document best practices. Same thing goes for filesystem structure and permissions. Providing examples for those, using "plone.recipe.command", for instance, would go a long way.
comment:55 Changed 8 years ago by rochael
BTW, folks, those interested in buildout+zeo+blob should subscribe to the launchpad issues created by klm above.
comment:56 in reply to: ↑ 54 Changed 8 years ago by klm
Replying to rochael:
Arranging for the blob directories to be the same is just a matter of reusing the configuration variable from one section to the other:
[zeoserver] recipe = plone.recipe.zope2zeoserver zope2-location = ${zope2:location} blob-storage = ${buildout:directory}/var/blobstorage zeo-address = 8100 eggs = ZODB3 ... [instance] ... blob-storage = ${zeoserver:blob-storage} ...
in fact, i do that in both my base.cfg and buildout.cfg instructions, using indirection through a [common] section. this makes ensuring synchronization between common settings for the server and clients intrinsice, like for eggs and products, which can be (are, in this case) numerous and cumbersome.
having the third [common] section has the added benefit of providing a handle that can be supplemented in derived sections. eg, it's turning out that in buildout.cfg i want my clients and server to use eggs and products that are used in the base.cfg, supplemented by entries for addons - i'm working on incorporating quills into my build. (i wish i knew how to refer to, eg, entries in the [server] section of base.cfg from the [server] section of buildout.cfg, but failed to determine how. that's one of the overwhelmingly many buildout nuances that i haven't got a clue how to track down.)
The recipes need not concern themselves with that, though we should obviously document best practices. Same thing goes for filesystem structure and permissions. Providing examples for those, using "plone.recipe.command", for instance, would go a long way.
is that the recipe used for the unified installer? or is there one? it would be nice to have some canonical recipes - and prominent pointers to them - for anyone looking for places to start in getting acquainted with crafting builds.
anyway, i digress all over the place - i'm very glad you're moving the zope2zeoserver recipe forward!
comment:57 Changed 8 years ago by rochael
Attention, everyone. Please test the recipes at branches:
and
comment:58 follow-up: ↓ 59 Changed 8 years ago by sunew
Just noticed: there's a wrong security declaration for fixAutoId in field.py. (Haven't got commit rights for plone yet ;) )
comment:59 in reply to: ↑ 58 Changed 8 years ago by witsch
comment:60 follow-up: ↓ 61 Changed 8 years ago by rochael
For those not so familiar with using development eggs, to use the branches above, check them out, say, into your src folder and declare them as development eggs:
$ cd src $ svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2zeoserver/branches/rochael-blobsupport plone.recipe.zope2zeoserver $ svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/branches/rochael-blobsupport plone.recipe.zope2instance
And in your [buildout] section:
develop = src/plone.recipe.zope2zeoserver src/plone.recipe.zope2instance (...)
comment:61 in reply to: ↑ 60 ; follow-up: ↓ 62 Changed 8 years ago by klm
Replying to rochael:
For those not so familiar with using development eggs, to use the branches above, check them out, say, into your src folder and declare them as development eggs:
$ cd src $ svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2zeoserver/branches/rochael-blobsupport plone.recipe.zope2zeoserver $ svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/branches/rochael-blobsupport plone.recipe.zope2instanceAnd in your [buildout] section:
develop = src/plone.recipe.zope2zeoserver src/plone.recipe.zope2instance (...)
though these measures should be unnecessary with hanno schlichting's commits to the trunk, i did notice a buildout trick that makes this kind of thing easier - worth noting for posterity.
at http://m.odul.us/2007/10/28/buildout-integrators, matt bowen uses an expression that gets alternative svn checkouts from the net, rather than having to download them manually. for these plone.receipes, you would set find-links entries to the svn address for the alternate repositories and append the name you want to use for the egg on the address - this is what i did in base.cfg:
find-links = [...] https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2zeoserver/branches/rochael-blobsupport#egg=plone.recipe.zope2zeoserver https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/branches/rochael-blobsupport#egg=plone.recipe.zope2instance
and add entries for those eggs to the eggs section:
eggs = elementtree plone.recipe.zope2instance plone.recipe.zope2zeoserver
with these changes (and no changes to the develop = section), the branch versions of the recipes will be automatically used instead of the cheese shop versions. (the branch ones are assigned version increments above the existing recipes, perhaps in lieu of having definite versions of their own.)
the branch versions should be unnecessary now, however, since Hanno Schlichting has checked all the fixes into the trunk. (it's great to see these things falling into place - yay!)
comment:62 in reply to: ↑ 61 Changed 8 years ago by klm
Replying to klm:
Replying to rochael:
For those not so familiar with using development eggs, to use the branches above, check them out, say, into your src folder and declare them as development eggs:
$ cd src $ svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2zeoserver/branches/rochael-blobsupport plone.recipe.zope2zeoserver $ svn co https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/branches/rochael-blobsupport plone.recipe.zope2instanceAnd in your [buildout] section:
develop = src/plone.recipe.zope2zeoserver src/plone.recipe.zope2instance (...)
i was mistaken, below, both about using find-links and about no longer needing to take special measures. just using find-links (at least, the way i was doing) doesn't work without a manual svn checkout, and hanno's trunk commits don't affect the released cheese shop versions - so you do need to fetch the revised recipes, at least until they're released. i recommend you get the trunk versions, since hanno's incorporated leo's fixes and added some.
cd src svn co http://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/trunk plone.recipe.zope2instance svn co http://svn.plone.org/svn/collective/buildout/plone.recipe.zope2zeoserver/trunk plone.recipe.zope2zeoserver
and include the develop = src/* lines that leo suggested in comment 60.
this will be unnecessary once the recipes changes are released, which may happen in a week or so.
though these measures should be unnecessary with hanno schlichting's commits to the trunk, i did notice a buildout trick that makes this kind of thing easier - worth noting for posterity.
at http://m.odul.us/2007/10/28/buildout-integrators, matt bowen uses an expression that gets alternative svn checkouts from the net, rather than having to download them manually. for these plone.receipes, you would set find-links entries to the svn address for the alternate repositories and append the name you want to use for the egg on the address - this is what i did in base.cfg:
find-links = [...] https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2zeoserver/branches/rochael-blobsupport#egg=plone.recipe.zope2zeoserver https://svn.plone.org/svn/collective/buildout/plone.recipe.zope2instance/branches/rochael-blobsupport#egg=plone.recipe.zope2instanceand add entries for those eggs to the eggs section:
eggs = elementtree plone.recipe.zope2instance plone.recipe.zope2zeoserverwith these changes (and no changes to the develop = section), the branch versions of the recipes will be automatically used instead of the cheese shop versions. (the branch ones are assigned version increments above the existing recipes, perhaps in lieu of having definite versions of their own.)
the branch versions should be unnecessary now, however, since Hanno Schlichting has checked all the fixes into the trunk. (it's great to see these things falling into place - yay!)
comment:63 follow-up: ↓ 64 Changed 8 years ago by witsch
quick update regarding the 2.5 backport: things seem to be working ttw and plone.app.blob's tests are passing as of 5 minutes ago; i haven't tried running CMFPlone and ATContentTypes tests yet, but we're getting closer... good night! :)
comment:64 in reply to: ↑ 63 ; follow-up: ↓ 65 Changed 8 years ago by witsch
Replying to witsch:
quick update regarding the 2.5 backport: things seem to be working ttw and plone.app.blob's tests are passing as of 5 minutes ago; i haven't tried running CMFPlone and ATContentTypes tests yet, but we're getting closer... good night! :)
i finally got around to try to run those tests as well (using the patches from the 3.0 buildout as a starting point), and after some minor fixes in r19071 almost all tests are passing now. the one from CMFPlone also fails ootb, but i still have to check the other one from ATCT...
comment:65 in reply to: ↑ 64 Changed 8 years ago by witsch
Replying to witsch:
i finally got around to try to run those tests as well (using the patches from the 3.0 buildout as a starting point), and after some minor fixes in r19071 almost all tests are passing now. the one from CMFPlone also fails ootb, but i still have to check the other one from ATCT...
looking into things again it turns out the two failing tests from ATCT,
Failure in test test_tool_auto_migration (Products.ATContentTypes.tests.test_atfile.TestSiteATFile) Traceback (most recent call last): File "/opt/zope/share/zope/Zope-2.9.8-final/lib/python/Testing/ZopeTestCase/profiler.py", line 98, in __call__ testMethod() File "/opt/zope/sprint/blob-2.5/parts/plone/ATContentTypes/tests/atcttestcase.py", line 298, in test_tool_auto_migration self.failUnlessEqual(obj.meta_type, self.meta_type) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 333, in failUnlessEqual raise self.failureException, \ AssertionError: 'ATBlob' != 'ATFile'
and
Failure in test test_title (Products.ATContentTypes.tests.test_atfile.TestATFileFields) Traceback (most recent call last): File "/opt/zope/share/zope/Zope-2.9.8-final/lib/python/Testing/ZopeTestCase/profiler.py", line 98, in __call__ testMethod() File "/opt/zope/sprint/blob-2.5/parts/plone/Archetypes/tests/test_baseschema.py", line 120, in test_title self.failUnless(isinstance(field.storage, AttributeStorage)) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 309, in failUnless if not expr: raise self.failureException, msg AssertionError
are really okay to have: for the first one the meta types obviously differ, but they also should. the second test expects the 'title' field to use an AttributeStorage, but plone.app.blob explicitly sets this field's storage to be an AnnotationStorage in plone/app/blob/content.py:37. if that line's commented out, the failure will go away as well.
so, it seems the backport to plone 2.5 was finished with r19071 (or actually even with r18913), at least as far as the automated tests are concerned. of course, it still needs to see some substantial testing in real projects, but assuming our coverage is good enough, there shouldn't be too many problems left, hopefully... :)
comment:66 Changed 8 years ago by forsberg
- Cc forsberg added
Trying out plone.app.blob, I ran into memory trouble. I opened up #7780 (or should I have reported it on this ticket instead?).
comment:67 follow-up: ↓ 68 Changed 8 years ago by forsberg
Minor problem: Every time I upload a new Blob, I get this on my console (running in fg mode):
Exception exceptions.OSError: (2, 'No such file or directory', '/tmp/tmp0hnTHq') in <bound method _TemporaryFileWrapper.__del__ of <closed file '<fdopen>', mode 'w+b' at 0xb1aac188>> ignored
comment:68 in reply to: ↑ 67 Changed 8 years ago by witsch
Replying to forsberg:
Minor problem: Every time I upload a new Blob, I get this on my console (running in fg mode):
yes, that's a known issue — see the faq. the good news is that leo made another fix for windows, which also solves this issue. so again, watch out for the next release... ;)
ps: opening another ticket was fine — this is already way too long anyway...
comment:69 follow-up: ↓ 70 Changed 8 years ago by garbas
anybody implementing image support?
comment:70 in reply to: ↑ 69 ; follow-up: ↓ 71 Changed 8 years ago by witsch
Replying to garbas:
anybody implementing image support?
yes, sune (brøndum wøller; "sunew" on irc) has looked into this and wanted to clean up & commit his changes. i'll be picking up his work (or eventually start my own version :)) soon, i.e. hopefully already next week, but in three weeks time at the latest. please also refer to #7822 for soon to come announcements... :)
comment:71 in reply to: ↑ 70 Changed 8 years ago by garbas
Replying to witsch:
Replying to garbas:
anybody implementing image support?
yes, sune (brøndum wøller; "sunew" on irc) has looked into this and wanted to clean up & commit his changes. i'll be picking up his work (or eventually start my own version :)) soon, i.e. hopefully already next week, but in three weeks time at the latest. please also refer to #7822 for soon to come announcements... :)
great ... well i'll try to be online tonight to catch you on irc. i already looked in latest code in trunk and it does not seems to be that hard to implement it also for other content types. well as you said i'll go to #7822 to discuss this more.
comment:73 follow-up: ↓ 74 Changed 8 years ago by klm
andy, i recently tried to create a fresh plone.app.blob based buildout, but blobs are failing in the resulting system. i described the problems in ticket [7986] as best i could, including attaching failures in tests of just plone.app.blob. i explored the issue a little, but don't have the time to uncover what's going on, and am worried i won't be able to migrate my existing sites to new installations...
ken
comment:74 in reply to: ↑ 73 ; follow-up: ↓ 75 Changed 8 years ago by klm
darn! my account of the problem is in ticket 7986, not changeset 7986. (i should have previewed before submitting the comment.)
Replying to klm:
andy, i recently tried to create a fresh plone.app.blob based buildout, but blobs are failing in the resulting system. i described the problems in ticket [7986] as best i could, including attaching failures in tests of just plone.app.blob. i explored the issue a little, but don't have the time to uncover what's going on, and am worried i won't be able to migrate my existing sites to new installations...
ken
comment:75 in reply to: ↑ 74 ; follow-ups: ↓ 76 ↓ 78 Changed 8 years ago by witsch
Replying to klm:
darn! my account of the problem is in ticket 7986, not changeset 7986.
:)
i've answered over there: don't use the ploneout-based buildout while 3.1 it in the making. i'll investigate things once it's been released and i've got some time to spare again... %)
comment:76 in reply to: ↑ 75 ; follow-up: ↓ 77 Changed 8 years ago by witsch
Replying to witsch:
i've answered over there: don't use the ploneout-based buildout while 3.1 it in the making. i'll investigate things once it's been released and i've got some time to spare again... %)
s/it in/is in/ (and yes, previewing helps :))
comment:77 in reply to: ↑ 76 Changed 8 years ago by klm
comment:78 in reply to: ↑ 75 Changed 8 years ago by witsch
Replying to witsch:
i've answered over there: don't use the ploneout-based buildout while 3.1 it in the making.
for the record, the buildout based on plone-coredev a.k.a ploneout is discontinued in favor of the new development buildout... if you want to help developing or simply use the "bleeding edge" version of the blob integration, please switch to the latter using
$ svn sw https://svn.plone.org/svn/plone/plone.app.blob/buildouts/devel
from within your buildout directory...
comment:79 Changed 8 years ago by witsch
comment:80 follow-up: ↓ 81 Changed 8 years ago by klm
the plone.app.blob checkout and build is broken, i believe because contentmigration is no longer available as an (external) checkout, instead it's available as the Products.contentmigration egg. (see the README-trunk.txt note.)
i believe what's necessary is to remove the Products.contentmigration entry from src/EXTERNALS.txt and add Products.contentmigration to the buildout.cfg [buildout] section's 'eggs' stanza.
i would try to do this on a branch, but don't have access to the main repository. i'll try copying it all to a repository i set up of my own, so i can make the changes and get past the checkout/update block - but i'm hoping that someone with plone.app.blob interest and checkin access can investigate and rectify promptly.
(with the general state of flux as things develop an transition to eggs, the lack of integration of plone.app.blob with the general plone build is increasingly dicey...)
comment:81 in reply to: ↑ 80 ; follow-up: ↓ 82 Changed 8 years ago by witsch
Replying to klm:
the plone.app.blob checkout and build is broken, i believe because contentmigration is no longer available as an (external) checkout, instead it's available as the Products.contentmigration egg.
thanks for pointing this out, ken. i've changed the external in r22200.
i believe what's necessary is to remove the Products.contentmigration entry from src/EXTERNALS.txt and add Products.contentmigration to the buildout.cfg [buildout] section's 'eggs' stanza.
no, i do want the development version in the development buildout of plone.app.blob. however, i still think you should use the egg-based buildout for testing or production, especially since i've recently started on image support, which could very well introduce new bugs... ;)
(with the general state of flux as things develop an transition to eggs, the lack of integration of plone.app.blob with the general plone build is increasingly dicey...)
like said above, it's probably a good idea to use the egg-based buildout unless you'd like to participate in the development of the blob integration.
comment:82 in reply to: ↑ 81 ; follow-up: ↓ 83 Changed 8 years ago by klm
Replying to witsch:
Replying to klm:
the plone.app.blob checkout and build is broken, i believe because contentmigration is no longer available as an (external) checkout, instead it's available as the Products.contentmigration egg.
thanks for pointing this out, ken. i've changed the external in r22200.
cool.
i believe what's necessary is to remove the Products.contentmigration entry from src/EXTERNALS.txt and add Products.contentmigration to the buildout.cfg [buildout] section's 'eggs' stanza.
no, i do want the development version in the development buildout of plone.app.blob. however, i still think you should use the egg-based buildout for testing or production, especially since i've recently started on image support, which could very well introduce new bugs... ;)
that's helpful, andi. i had shifted to basing my cluster build on the simple zeo recipe in the plone.app.blob product and pypi entries. apparently that's more along the lines of the development buildout - the plone-3.x egg-based buildout defers much more to the conventional builds, and so is much better for my purposes.
(with the general state of flux as things develop an transition to eggs, the lack of integration of plone.app.blob with the general plone build is increasingly dicey...)
like said above, it's probably a good idea to use the egg-based buildout unless you'd like to participate in the development of the blob integration.
aok - thanks! i've adopted the plone-3.x build and it works. i'm hitting a problem migrating my existing blob-containing builds, which i'll detail in a separate entry in this issue...
comment:83 in reply to: ↑ 82 Changed 8 years ago by klm
Replying to klm:
Replying to witsch:
like said above, it's probably a good idea to use the egg-based buildout unless you'd like to participate in the development of the blob integration.
aok - thanks! i've adopted the plone-3.x build and it works. i'm hitting a problem migrating my existing blob-containing builds, which i'll detail in a separate entry in this issue...
it turns out that i'm unsure about the cause of an unpickling problem (seeking Products.ATBlobFile.interfaces.IFileUpload), and suspect it's separate from plone.app.blob. sorry about the false alarm...
comment:84 in reply to: ↑ 15 ; follow-up: ↓ 85 Changed 8 years ago by grahamperrin
Replying to witsch:
you can also try to point your browser to .../@@migrate-files-to-blobs to migrate all you existing File content to Blob instances
Reading that 2007 comment alongside more recent http://plone.org/products/plone.app.blob and http://pypi.python.org/pypi/plone.app.blob I assume that
/@@migrate-files-to-blobs
is superseded by
/@@blob-migration
— please, can we confirm? Thanks.
comment:85 in reply to: ↑ 84 Changed 8 years ago by witsch
Replying to grahamperrin:
— please, can we confirm? Thanks.
yes, and no. while the latest version (i.e. trunk, with a new release pending) still has /@@blob-migration (migrating all your "File" content to a new content type "Blob"), you most likely want to use /@@blob-file-migration ("File" to blob-based "File") and /@@blob-image-migration ("Image" to blob-based "Image") instead. this will not introduce an additional content type, but "simply" convert your existing "File" and "Image" content to use blobs (with hopefully no other noticeable changes :)). this is more in line with #7822 as well, but please note that in order to use these, you'll have to apply the "replacement" profile first.
and yes, the docs are outdated in that regard, but the newer version hasn't been properly released yet, either. it will be soon, though.
comment:86 Changed 7 years ago by hannosch
- Milestone changed from 4.0 to Ongoing
This ticket isn't really tied to any release but an ongoing discussion.
For Plone 4.0 a real PLIP will be required to get this into the core.
comment:87 Changed 7 years ago by jteeters
Using instruction on the product page: http://plone.org/products/plone.app.blob with buildout, I seem to got it working with Plone 3.2.1 (current latest release).
By "working" I mean, when I add a new item of type "File" in Plone, the uploaded file is stored in the var/blobstorage directory.
However, when I add an attachment (inside an item of type RichDocument which uses SimpleAttachment), the attachment is not stored in blobstorage.
Three questions:
- Is it possible to get SimpleAttachment to use blobstorage? If so, what do I need to do?
- Is the patch to file products/ATContentTypes/content/file.py (given in comment dated "07/20/07 23:50:07") no longer applicable? I tried making those changes but got the following error when starting the instance:
ImportError: No module named ATBlobField.field
- What is the general state of this product? It seems exactly what I need, but there have not been any posts here for a while. Is it mature enough that it's likely to work well or is there some other product I should consider for storing blobs on the file system and not in the zope db? This is for a production site, but a non-critical application.
Thanks!
comment:88 follow-up: ↓ 89 Changed 7 years ago by hoss
I'm unable to get things working with plone 3.2.1 and r24112 of plone.app.blob.
See http://plone.org/products/plone.app.blob/issues/18/view
Basically, it throws an exception because of a bogus reference to 'targetpath'.
I had tried the 1.0b2 version of the pypi egg for plone.app.blob, but that didn't work either, which is why I went to the svn trunk (r26015/r24112 as of this post).
I get the distinct impression that plone.app.blob has not kept pace with recent plone core updates. The last concrete confirmed version support was plone 3.0.4.
Please advise. Thanks.
comment:89 in reply to: ↑ 88 ; follow-up: ↓ 90 Changed 7 years ago by klm
Replying to hoss:
I'm unable to get things working with plone 3.2.1 and r24112 of plone.app.blob.
See http://plone.org/products/plone.app.blob/issues/18/view
Basically, it throws an exception because of a bogus reference to 'targetpath'.
I had tried the 1.0b2 version of the pypi egg for plone.app.blob, but that didn't work either, which is why I went to the svn trunk (r26015/r24112 as of this post).
I get the distinct impression that plone.app.blob has not kept pace with recent plone core updates. The last concrete confirmed version support was plone 3.0.4.
for what it's worth, i have been using plone.app.blob with the most recent version of plone short of plone 3.2. i recently updated my buildout and it seems to be working fine, winding up using plone 3.1.7. (i am under a delivery deadline, and this bug report will dissuade me from trying without the <3.2 constraint until afterwards.)
my buildout plone section looks like this:
[plone] recipe = plone.recipe.plone >=3.1.1,<3.2dev
Please advise. Thanks.
i'm sorry i can't offer more than evidence that it does work with plone 3.1.7, and a suggestion to settle for that until someone takes responsibility for addressing plone 3.2 compatability.
comment:90 in reply to: ↑ 89 Changed 7 years ago by hoss
Replying to klm:
(i am under a delivery deadline, and this bug report will dissuade me from trying without the <3.2 constraint until afterwards.)
Thanks for the quick reply.
Yeah, if you've got anything serious you're doing, I'd steer clear of 3.2 until this gets addressed. I may have just had something askew that kept plone.app.blob from working, but I was using as 'clean' of a test as I could manage.
My current choice, it would seem, is to either fall back to <3.2, or just go ahead with my deployment without plone.app.blob. I'm leaning towards the latter, since there seems to be good support for migrating instances of the stock objects to blob storage. I'm not going to be accruing a ton of content right away, so I think I can afford to wait a bit until p.a.b catches up with plone.
comment:92 Changed 7 years ago by witsch
fyi, while the new beta fixes a number of issues, it's also a bit of a "snapshot" release i wanted to get out because of two reasons: firstly issue 7 was starting to cause serious trouble in a couple of installations — the last release was clearly showing how outdated it had become.
in addition, i won't be available for about two weeks. i'll continue to work on the package afterwards (as it'll be added to a production deployment soon), but didn't want to hold back the recent progress before that break. there'll also be a hopefully comprehensive blog post soon, but for now there's something far more important to do... :)
comment:96 Changed 6 years ago by witsch
- Priority changed from major to trivial
comment:103 Changed 6 years ago by witsch
comment:105 Changed 6 years ago by kleist
Shouldn't we close this one?
comment:106 Changed 6 years ago by limi
- Status changed from assigned to closed
- Priority changed from trivial to n/a
- Resolution set to fixed
- Type changed from Feature Request to PLIP
Yes, we should — witsch also said that we should track any other issues separately, and I agree.
since i've started working on this during the piktipi sprint in potsdam and continued during the last couple of days (sprinting remotely with the folks in boston), i'll take it... :) well, actually the ticket was intended to provide some sort of progress log for myself... :)