Quantcast

Uploading large files

classic Classic list List threaded Threaded
12 messages Options
Ep
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Uploading large files

Ep
I want to create a content upload service that sha1 hashes the content and stores it to a compressed file. Some files may be quite large (2GB+ uncompressed). Would unfiltered be a good fit for implementing this type of service?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uploading large files

n8han
Administrator
Yes, assuming you intend to use http. Is the remote client a regular browser or something custom?

Nathan

On 07/21/2011 04:15 PM, Ep [via Databinder] wrote:
I want to create a content upload service that sha1 hashes the content and stores it to a compressed file. Some files may be quite large (2GB+ uncompressed). Would unfiltered be a good fit for implementing this type of service?


If you reply to this email, your message will be added to the discussion below:
http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608364.html
To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uploading large files

Dustin Whitney
I've been curious about this myself.  I'd like to hear what your experience is.

Dustin

On Thu, Jul 21, 2011 at 4:17 PM, n8han [via Databinder] <[hidden email]> wrote:
Yes, assuming you intend to use http. Is the remote client a regular browser or something custom?

Nathan


On 07/21/2011 04:15 PM, Ep [via Databinder] wrote:
I want to create a content upload service that sha1 hashes the content and stores it to a compressed file. Some files may be quite large (2GB+ uncompressed). Would unfiltered be a good fit for implementing this type of service?


If you reply to this email, your message will be added to the discussion below:
http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608364.html
To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.




If you reply to this email, your message will be added to the discussion below:
http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608375.html
To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

Ep
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uploading large files

Ep
In reply to this post by n8han
Something custom. Basically something like SillyStore but the key is a sha1 of the bytes...and save/load the compressed files to/from disk. How to sha1 and compress as content is streamed to disk? How to decompress and stream to client? etc.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uploading large files

chris_lewis
Now you're outside the realm of unfiltered, but ...

Hashes (like sha) are based on the full contents of a file, so you can't resolve its hash without its full contents. You could investigate calculating hashes incrementally. That is, for every 100mb, calculate a sha and aggregate the hashes per each chunk. If your goal is to generate the expected <name>.sha, then this option is out.

I'm not sure about the compression part, but I imagine you can't do it, at least not well, incrementally. I'd be in input from others though!

-chris

On 7/21/11 4:31 PM, Ep [via Databinder] wrote:
Something custom. Basically something like SillyStore but the key is a sha1 of the bytes...and save/load the compressed files to/from disk. How to sha1 and compress as content is streamed to disk? How to decompress and stream to client? etc.


If you reply to this email, your message will be added to the discussion below:
http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608417.html
To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uploading large files

chris_lewis
In reply to this post by Ep
On 7/21/11 4:38 PM, Chris Lewis wrote:
Now you're outside the realm of unfiltered, but ...

Hashes (like sha) are based on the full contents of a file, so you can't resolve its hash without its full contents. You could investigate calculating hashes incrementally. That is, for every 100mb, calculate a sha and aggregate the hashes per each chunk. If your goal is to generate the expected <name>.sha, then this option is out.

I'm not sure about the compression part, but I imagine you can't do it, at least not well, incrementally. I'd be in input from others though!
Ahem - "interested in."

-chris

On 7/21/11 4:31 PM, Ep [via Databinder] wrote:
Something custom. Basically something like SillyStore but the key is a sha1 of the bytes...and save/load the compressed files to/from disk. How to sha1 and compress as content is streamed to disk? How to decompress and stream to client? etc.


If you reply to this email, your message will be added to the discussion below:
http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608417.html
To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.


Ep
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uploading large files

Ep
In reply to this post by chris_lewis
May just hash then compress the file on the client (or could do it on the server as well once the file is uploaded). So it then becomes...how to move large file via http with unfiltered and what are the size limits?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uploading large files

n8han
Administrator
You are going to have any problem with size limits from unfiltered. You can choose to handle a blocking InputStream coming from a filter, or the non-blocking interfaces that Netty gives you.

If you intend to support many simultaneous uploads and don't need to work through the servlet API, Netty is probably a better choice. But it's worth building prototypes of each.

Nathan

On 07/21/2011 04:43 PM, Ep [via Databinder] wrote:
May just hash then compress the file on the client (or could do it on the server as well once the file is uploaded). So it then becomes...how to move large file via http with unfiltered and what are the size limits?


If you reply to this email, your message will be added to the discussion below:
http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608458.html
To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

Ep
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uploading large files

Ep
Not restricted to the servlet api and will give Netty a try. But with the servlet route, would you recommend the upload plugin?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uploading large files

n8han
Administrator
Not really. That plugin handles multipart posts and since you're in control of the client you can do a simple PUT like the silly store example, with less overhead and complexity.

On 07/21/2011 05:05 PM, Ep [via Databinder] wrote:
Not restricted to the servlet api and will give Netty a try. But with the servlet route, would you recommend the upload plugin?


If you reply to this email, your message will be added to the discussion below:
http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608516.html
To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uploading large files

Dustin Whitney
If you use Jetty, you'll need to set the post size limit higher.  Here is how I do it - not sure if anyone has a better way:

for(hs <- server.underlying.getHandlers) hs match {
      case handlers:ContextHandlerCollection =>
        for(h <- handlers.getHandlers) h match {
          case s:ServletContextHandler =>
            s.setMaxFormContentSize(5242880) //5mb limit
        }
    }

On Thu, Jul 21, 2011 at 5:13 PM, n8han [via Databinder] <[hidden email]> wrote:
Not really. That plugin handles multipart posts and since you're in control of the client you can do a simple PUT like the silly store example, with less overhead and complexity.


On 07/21/2011 05:05 PM, Ep [via Databinder] wrote:
Not restricted to the servlet api and will give Netty a try. But with the servlet route, would you recommend the upload plugin?


If you reply to this email, your message will be added to the discussion below:
http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608516.html
To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.




If you reply to this email, your message will be added to the discussion below:
http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608540.html
To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uploading large files

n8han
Administrator
I don't think that should apply to the request body, cause that would be really lame! I think it is just if you are trying to get it out of a parameter, if you're using a multipart post.

http://permalink.gmane.org/gmane.comp.java.jetty.support/13510

Nathan

On 07/22/2011 09:32 AM, Dustin Whitney [via Databinder] wrote:
If you use Jetty, you'll need to set the post size limit higher.  Here is how I do it - not sure if anyone has a better way:

for(hs <- server.underlying.getHandlers) hs match {
      case handlers:ContextHandlerCollection =>
        for(h <- handlers.getHandlers) h match {
          case s:ServletContextHandler =>
            s.setMaxFormContentSize(5242880) //5mb limit
        }
    }

On Thu, Jul 21, 2011 at 5:13 PM, n8han [via Databinder] <[hidden email]> wrote:
Not really. That plugin handles multipart posts and since you're in control of the client you can do a simple PUT like the silly store example, with less overhead and complexity.


On 07/21/2011 05:05 PM, Ep [via Databinder] wrote:
Not restricted to the servlet api and will give Netty a try. But with the servlet route, would you recommend the upload plugin?


If you reply to this email, your message will be added to the discussion below:
http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608516.html
To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.




If you reply to this email, your message will be added to the discussion below:
http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6608540.html
To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.




If you reply to this email, your message will be added to the discussion below:
http://databinder.3617998.n2.nabble.com/Uploading-large-files-tp6608364p6610475.html
To start a new topic under Unfiltered, email [hidden email]
To unsubscribe from Unfiltered, click here.

Loading...