I want to create a content upload service that sha1 hashes the content and stores it to a compressed file. Some files may be quite large (2GB+ uncompressed). Would unfiltered be a good fit for implementing this type of service?
|
Administrator
|
Yes, assuming you intend to use http. Is the remote client a regular
browser or something custom?
Nathan On 07/21/2011 04:15 PM, Ep [via Databinder] wrote: I want to create a content upload service that sha1 hashes the content and stores it to a compressed file. Some files may be quite large (2GB+ uncompressed). Would unfiltered be a good fit for implementing this type of service? |
I've been curious about this myself. I'd like to hear what your experience is.
Dustin On Thu, Jul 21, 2011 at 4:17 PM, n8han [via Databinder] <[hidden email]> wrote: Yes, assuming you intend to use http. Is the remote client a regular browser or something custom? |
In reply to this post by n8han
Something custom. Basically something like SillyStore but the key is a sha1 of the bytes...and save/load the compressed files to/from disk. How to sha1 and compress as content is streamed to disk? How to decompress and stream to client? etc.
|
Now you're outside the realm of unfiltered, but ...
Hashes (like sha) are based on the full contents of a file, so you can't resolve its hash without its full contents. You could investigate calculating hashes incrementally. That is, for every 100mb, calculate a sha and aggregate the hashes per each chunk. If your goal is to generate the expected <name>.sha, then this option is out. I'm not sure about the compression part, but I imagine you can't do it, at least not well, incrementally. I'd be in input from others though! -chris On 7/21/11 4:31 PM, Ep [via Databinder] wrote: Something custom. Basically something like SillyStore but the key is a sha1 of the bytes...and save/load the compressed files to/from disk. How to sha1 and compress as content is streamed to disk? How to decompress and stream to client? etc. |
In reply to this post by Ep
On 7/21/11 4:38 PM, Chris Lewis wrote:
Now you're outside the realm of unfiltered, but ...Ahem - "interested in."
|
In reply to this post by chris_lewis
May just hash then compress the file on the client (or could do it on the server as well once the file is uploaded). So it then becomes...how to move large file via http with unfiltered and what are the size limits?
|
Administrator
|
You are going to have any problem with size limits from unfiltered.
You can choose to handle a blocking InputStream coming from a
filter, or the non-blocking interfaces that Netty gives you.
If you intend to support many simultaneous uploads and don't need to work through the servlet API, Netty is probably a better choice. But it's worth building prototypes of each. Nathan On 07/21/2011 04:43 PM, Ep [via Databinder] wrote: May just hash then compress the file on the client (or could do it on the server as well once the file is uploaded). So it then becomes...how to move large file via http with unfiltered and what are the size limits? |
Not restricted to the servlet api and will give Netty a try. But with the servlet route, would you recommend the upload plugin?
|
Administrator
|
Not really. That plugin handles multipart posts and since you're in
control of the client you can do a simple PUT like the silly store
example, with less overhead and complexity.
On 07/21/2011 05:05 PM, Ep [via Databinder] wrote: Not restricted to the servlet api and will give Netty a try. But with the servlet route, would you recommend the upload plugin? |
If you use Jetty, you'll need to set the post size limit higher. Here is how I do it - not sure if anyone has a better way:
for(hs <- server.underlying.getHandlers) hs match { case handlers:ContextHandlerCollection => for(h <- handlers.getHandlers) h match { case s:ServletContextHandler => s.setMaxFormContentSize(5242880) //5mb limit } } On Thu, Jul 21, 2011 at 5:13 PM, n8han [via Databinder] <[hidden email]> wrote: Not really. That plugin handles multipart posts and since you're in control of the client you can do a simple PUT like the silly store example, with less overhead and complexity. |
Administrator
|
http://permalink.gmane.org/gmane.comp.java.jetty.support/13510 Nathan On 07/22/2011 09:32 AM, Dustin Whitney [via Databinder] wrote: If you use Jetty, you'll need to set the post size limit higher. Here is how I do it - not sure if anyone has a better way: |
Free forum by Nabble | Edit this page |